Local Minimum Using SGD, ADAM, OGR Gradient Descent Optimizers

Finding a local minimum, possibly using the Wolfram Language FindMinimum function, is a basic numerical problem at the heart of neural network training. Finding a local minimum is usually done by calculating gradients at neighboring initial points and descending along them (gradient descent). The object is then to optimize convergence speed. Calculated gradients are often inaccurate (simulated here by adding Gaussian noise), suggesting the use of some averaging procedure (e.g. the momentum method).

This Demonstration allows testing of five methods on five popular two-dimensional functions (with minimum shifted to ). SGD stands for stochastic gradient descent and ADAM for Adaptive Moment Estimation.

You can draw trajectories in 2D or 3D and evaluate on a lattice of starting points. Similar basic standard methods include vanilla SGD and momentum. However, these are very inefficient and require a tiny learning rate. There is also the very popular ADAM method that gives much better results.

Two additional methods are also considered: OGR (online gradient regression, see arXiv:1901.11457) and using a Newton-step method with second derivative to estimate the observed sequence of gradients. Diagonal Hessian variant cdOGR models consider a separate parabola for each canonical direction.