Gradient

In multiple dimensions, the Gradient is the vector of partial derivatives along each dimension
The slope in any direction is the dot product of the direction with the gradient. The direction of steepest descent is the negative gradient

Types of gradients

Numerical gradient

• approximate, slow, easy to write
• Calculate by finding slope for each value, add small value and see difference

Analytic gradient

• exact, fast, error-prone
• Use analytic expression to calculate gradient