Supervised Learning

Memoriser vs Learner

A perfect score on the examples can still fail on brand-new ones. Turn the wiggle knob and find the curve that learns the pattern, not the noise.

The filled dots are days I get to study. The hollow dots are secret test days. Drag the wiggle level and try to find the best one!

study dots test dots the curve

Tap empty space to add a study dot · drag a dot to move it · tap a dot to remove it · dashed line = the real pattern

Challenges

Find the sweet spot: the level with the smallest red test error.
Make a memoriser: level 9 or more, great on study dots, bad on test dots. Try fewer dots.
Tame it: give it 30+ examples and keep the wiggle at 9 or more. Overfitting shrinks!

What just happened?

For grown-ups: what's really going on

This is polynomial regression of degree d (wiggle level) fit by least squares: solve the normal equations (XᵀX + λI) w = Xᵀy with a tiny ridge term λ = 10⁻⁶ for stability. Inputs are scaled to [-1, 1] and a Chebyshev basis is used so the matrix stays well-conditioned.

  • Training error only falls as d grows (nested models), reaching zero once d + 1 ≥ n. Test (generalisation) error is U-shaped.
  • The bias-variance trade-off: low degree means high bias (underfitting); high degree means high variance (overfitting), the curve changes a lot with each new sample of noise.
  • Errors shown are RMSE. Picking the level from the test dots is a form of validation; a real project also keeps a final untouched test set, or uses cross-validation.
  • Remedies: more data, regularisation (larger λ), simpler models, early stopping.

Further reading: An Introduction to Statistical Learning, ch. 2 and 5 · Hastie et al., Elements of Statistical Learning, ch. 7.