Learning Rate Race
A ball rolls downhill to find the lowest spot. How big should each step be? Too tiny and it crawls. Too big and it flies away!
Drag the ball anywhere (or tap the map), pick a step size, then press Roll.
Play: roll a ball
Watch it learn: the race
Four balls, one hill, four step sizes. Press Race! and watch who gets home first.
Challenges
Each one loads the race above. When you win, the chart marks the exact step with a dashed line.
How it works
Feel the slope
The ball checks which way is downhill and how steep it is right where it stands.
Take a step
It moves downhill. The step size decides how far: steeper ground means a longer hop.
Repeat
Look, step, look, step. Every step should put it a little lower.
Add some speed
A heavy ball keeps some of its old speed, so it can zoom along valleys and roll over small dips.
What just happened?
- The ball always stepped downhill. The only thing we changed was how big the step was.
- Tiny steps are safe but slow. Giant steps overshoot, bounce, or even fly away.
- In a narrow valley plain balls zig-zag. A heavy ball, with a bit of speed, goes straight along it.
- Starting big and getting smaller lets a ball move fast first, then settle gently.
For grown-ups: what's really going on
The landscape is a loss function L(w); the ball position is the parameter vector w, and each step is gradient descent: w ← w − η∇L(w).
- Learning rate η: the slider. On a quadratic with curvature λ the iteration is stable only for
η < 2/λ; above that each step overshoots by more than it started (divergence). That is the "flew away" ball. Below it, convergence speed is roughly1 − ηλper step, so a tiny η crawls. - Narrow valley: the demo valley has curvature about 0.1 along the floor and 3 across it (ill-conditioned, condition number ~30). η is capped by the steep direction, so the flat direction is slow, and large η zig-zags.
- Momentum (heavy ball):
v ← βv − η∇L; w ← w + vwith β = 0.9 to 0.95. Gradients that agree accumulate (speed along the valley); ones that flip sign cancel (less zig-zag). It can also carry the ball through shallow local minima, as in the bumpy field. - Schedules: "Slowing down" uses
η_t = η₀ / (1 + 0.04t)(inverse-time decay). Large early steps for progress, small late steps to settle; cosine and step decay are common alternatives. - Adam-lite: keeps running averages of the gradient (
m) and its square (v) with bias correction, steppingη·m̂/(√v̂+ε). Each parameter gets its own effective step size. Real Adam is the same idea. - SGD noise: here gradients are exact (full batch). With mini-batches the gradient is noisy; the noise helps escape sharp minima but means a constant η never fully settles, hence decay schedules.
- Real losses have far more dimensions, with saddle points (flat but not minima) more common than bad local minima. The step-size lesson is the same: tune η first.
- Further reading: Goh, Why Momentum Really Works · Ruder, An overview of gradient descent optimization algorithms.