Stochastic Gradient Descent

Stochastic gradient descent is an optimization algorithm for finding the minimum or maximum of an objective function. In this Demonstration, stochastic gradient descent is used to learn the parameters (intercept and slope) of a simple regression problem. Using "contour plot", the likelihood function of the parameters is shown as a contour plot. The blue point gives the actual parameters while the red point shows the iterates of the stochastic gradient function. Selecting "regression" shows the data points, the actual regression line in green, and the iterative regression line in red as determined by the algorithm.



  • [Snapshot]
  • [Snapshot]
  • [Snapshot]


Consider a simple regression function . The parameters of the function are the slope and the intercept . The errors associated with each data point are assumed to be independent and normally distributed with variance . Given a sample data point , the likelihood of the parameters is specified as . Maximum-likelihood estimation uses the joint-likelihood function of all the data points to learn the parameters of the regression line. Stochastic gradient descent uses each data point to iteratively update the estimated parameters of the algorithm by traversing the likelihood surface in the direction of the negative gradient of the likelihood of each point. The amount of travel in the direction of the point gradient is specified by the learning parameter of the algorithm. The algorithm is as follows.
1. Choose a learning parameter , an initial estimate of the parameters .
2. Produce a random permutation of the data points.
3. For each point, compute .
4. Repeat steps 2 and 3 until some convergence criterion is met.
[1] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed., New York: Springer, 2001.
[2] Wikipedia, "Stochastic Gradient Descent." (Feb 28, 2011) http://en.wikipedia.org/wiki/Stochastic_gradient_descent.
    • Share:

Embed Interactive Demonstration New!

Just copy and paste this snippet of JavaScript code into your website or blog to put the live Demonstration on your site. More details »

Files require Wolfram CDF Player or Mathematica.

Mathematica »
The #1 tool for creating Demonstrations
and anything technical.
Wolfram|Alpha »
Explore anything with the first
computational knowledge engine.
MathWorld »
The web's most extensive
mathematics resource.
Course Assistant Apps »
An app for every course—
right in the palm of your hand.
Wolfram Blog »
Read our views on math,
science, and technology.
Computable Document Format »
The format that makes Demonstrations
(and any information) easy to share and
interact with.
STEM Initiative »
Programs & resources for
educators, schools & students.
Computerbasedmath.org »
Join the initiative for modernizing
math education.
Step-by-Step Solutions »
Walk through homework problems one step at a time, with hints to help along the way.
Wolfram Problem Generator »
Unlimited random practice problems and answers with built-in step-by-step solutions. Practice online or make a printable study sheet.
Wolfram Language »
Knowledge-based programming for everyone.
Powered by Wolfram Mathematica © 2018 Wolfram Demonstrations Project & Contributors  |  Terms of Use  |  Privacy Policy  |  RSS Give us your feedback
Note: To run this Demonstration you need Mathematica 7+ or the free Mathematica Player 7EX
Download or upgrade to Mathematica Player 7EX
I already have Mathematica Player or Mathematica 7+