Gradient descent

Machine learning models learn by adjusting their parameters to reduce error. The mechanism behind most of this adjustment is gradient descent: a way of finding the minimum of a function by repeatedly stepping downhill, in the direction of steepest descent.

The loss curve

Picture a single parameter x and a loss function f(x) that measures how much error the model makes for that value of x. The goal is to find the value of x where the loss is lowest.

The function used here is a one-dimensional asymmetric double well:

f(x) = 0.08 * (x^2 - 4)^2 + 0.15 * x

It has two valleys rather than one. The small tilt from the 0.15 * x term makes the left valley slightly deeper than the right one, so the two minima are not equal. This shape is deliberate: a single bowl can only show one minimum, but a double well can show the difference between a local minimum and the global minimum.

The gradient

In one dimension the gradient is just the derivative f'(x), the slope of the curve. To descend you step in the opposite direction of the slope:

f'(x) = 0.32 * x * (x^2 - 4) + 0.15

The parameter update at each step:

x  <-  x - alpha * f'(x)

where alpha (the learning rate) is a small positive number that controls how large each step is. When the slope is positive the point moves left; when it is negative the point moves right. Either way it moves toward lower loss.

Local versus global minima

Setting f'(x) to zero gives three points where the slope is flat:

x ≈ -2.056   global minimum   f ≈ -0.304   the deeper, left valley
x ≈ +0.118   local maximum    f ≈ +1.289   the ridge between the valleys
x ≈ +1.939   local minimum    f ≈ +0.296   the shallower, right valley

A minimum is global if no other point has a lower loss, and local if it is the lowest only within its immediate neighborhood. Here the left valley is the global minimum and the right valley is a local minimum that is higher than the global one.

Why the starting point matters

Every step points downhill: the update always moves against the slope. With a small learning rate that means the point can never cross the ridge at x ≈ 0.118 that separates the two valleys. Whichever side of the ridge you start on is the valley you end up in.

Start the demo on the right, at x = 2.4, and run it at the default learning rate. The point rolls left and down until the slope flattens at x ≈ 1.939, then stops. That is the local minimum. It settles there even though a deeper, better minimum exists in the left valley, because at a small learning rate each step is too short to climb the ridge, and the loss falls at every single step.

Now switch to Start left, at x = -2.4, and run again. This time the point settles at x ≈ -2.056, the global minimum.

The learning rate changes all of this. A step points downhill, but a large step can land uphill by overshooting the valley floor. Raise the slider past the careful range and the point starts to bounce, landing higher than it left. Push it further and the bounces grow until the steps blow up and the demo reports Diverged. At the highest rates the point can even be flung across the ridge it could never walk over. Smooth, guaranteed descent is a property of small steps, not of gradient descent itself.

This is the central limitation the demo is built to show: gradient descent finds a minimum near where it begins, not necessarily the best minimum overall. It is why where you initialize the parameters matters, and why techniques such as random restarts, momentum, and stochastic noise exist. Each is a way to give the optimizer a chance to escape a local minimum and find a deeper one.

What the demo shows

The grey curve is the loss f(x). The red dot is the current value of x. The red arrow shows the descent direction, which is the sign of the negative slope. The dashed grey trail records every previous position. The filled black dot marks the global minimum and the hollow black ring marks the local minimum. Use Play to run the optimization, Step to advance one iteration at a time and watch the readouts, and Start left or Start right to choose which valley the point begins in. Drag the red point while paused to set the starting position. The learning rate slider sets the step size: small rates descend smoothly, large ones overshoot, oscillate, and can diverge.

The loss curve is a hand-picked toy function, one dimensional so the walk stays visible. Everything else is computed from it for real: the slope, every step of the update rule, the settling, and the divergence at high learning rates all come from evaluating f and its derivative live, nothing is scripted.