Gradient Descent

# Vanilla gradient descent

while True:
	weights_grad = evaluate_gradient(loss_fun, data, weights)
	weights += - step_size * weights_grad

step_size/learning rate is Hyperparameters

L = \frac{1}{N} \sum_{i} L_{i} (f (x_{i}, W), y_{i}) + λ R (W)

\nabla_{W} L = \frac{1}{N} \sum_{i} \nabla_{W} L_{i} (f (x_{i}, W), y_{i}) + λ \nabla_{W} R (W)

Full sum expensive when N is large!

Approximate sum using a minibatch of examples 32/64/128 commons: Stochastic Gradient Descent