Neural Networks

Learning Rate Race

A ball rolls downhill to find the lowest spot. How big should each step be? Too tiny and it crawls. Too big and it flies away!

Drag the ball anywhere (or tap the map), pick a step size, then press Roll.

Play: roll a ball

Yellow = low (good) Purple = high (bad) Dashed ring = start

Watch it learn: the race

Four balls, one hill, four step sizes. Press Race! and watch who gets home first.

How high up the hill each ball is, step by step (lower is better)
Challenge doneBig dot = reached the bottomx = flew away

Challenges

Each one loads the race above. When you win, the chart marks the exact step with a dashed line.

How it works

Feel the slope

The ball checks which way is downhill and how steep it is right where it stands.

Take a step

It moves downhill. The step size decides how far: steeper ground means a longer hop.

Repeat

Look, step, look, step. Every step should put it a little lower.

Add some speed

A heavy ball keeps some of its old speed, so it can zoom along valleys and roll over small dips.

What just happened?

For grown-ups: what's really going on

The landscape is a loss function L(w); the ball position is the parameter vector w, and each step is gradient descent: w ← w − η∇L(w).

  • Learning rate η: the slider. On a quadratic with curvature λ the iteration is stable only for η < 2/λ; above that each step overshoots by more than it started (divergence). That is the "flew away" ball. Below it, convergence speed is roughly 1 − ηλ per step, so a tiny η crawls.
  • Narrow valley: the demo valley has curvature about 0.1 along the floor and 3 across it (ill-conditioned, condition number ~30). η is capped by the steep direction, so the flat direction is slow, and large η zig-zags.
  • Momentum (heavy ball): v ← βv − η∇L; w ← w + v with β = 0.9 to 0.95. Gradients that agree accumulate (speed along the valley); ones that flip sign cancel (less zig-zag). It can also carry the ball through shallow local minima, as in the bumpy field.
  • Schedules: "Slowing down" uses η_t = η₀ / (1 + 0.04t) (inverse-time decay). Large early steps for progress, small late steps to settle; cosine and step decay are common alternatives.
  • Adam-lite: keeps running averages of the gradient (m) and its square (v) with bias correction, stepping η·m̂/(√v̂+ε). Each parameter gets its own effective step size. Real Adam is the same idea.
  • SGD noise: here gradients are exact (full batch). With mini-batches the gradient is noisy; the noise helps escape sharp minima but means a constant η never fully settles, hence decay schedules.
  • Real losses have far more dimensions, with saddle points (flat but not minima) more common than bad local minima. The step-size lesson is the same: tune η first.
  • Further reading: Goh, Why Momentum Really Works · Ruder, An overview of gradient descent optimization algorithms.