Learning Rate

Stochastic Gradient Descent, AdaGrad, Adam all have learning rate as a hyperparameter

Learning Rate Decay

Reduce learning rate at a few fixed points.

• E.g. for resNets, Multiply LR by 0,1 after epochs 30, 60, and 90.
  Examples of decay:
• Cosine ${\alpha }_t=\frac{1}{2}{\alpha }_0(1+\cos (t\pi /T))$
  • Two Hyperparameters
  • No new hyperparameters
    
• Linear ${\alpha }_t={\alpha }_0(1-t/T)$
  
• Inverse sqrt ${\alpha }_t={\alpha }_0/\sqrt{t}$
  • Not a lot of time in high learning rate
  • A lot of time in low learning rate
    
• constant ${\alpha }_t={\alpha }_0$
  • Works well most of the time
    

Linear warmup