Memoriser vs Learner
A perfect score on the examples can still fail on brand-new ones. Turn the wiggle knob and find the curve that learns the pattern, not the noise.
The filled dots are days I get to study. The hollow dots are secret test days. Drag the wiggle level and try to find the best one!
Tap empty space to add a study dot · drag a dot to move it · tap a dot to remove it · dashed line = the real pattern
Challenges
What just happened?
- Too stiff a curve misses the pattern: it is wrong on study dots and test dots. That is underfitting.
- Too wiggly a curve hits every study dot but wobbles wildly between them. That is memorising.
- The honest score is on dots the model never studied, the hollow test dots.
- The best wiggle level is where the test error is lowest, the bottom of the U.
- More examples or less noise make the wiggly curve behave better.
For grown-ups: what's really going on
This is polynomial regression of degree d (wiggle level) fit by least squares: solve the normal equations (XᵀX + λI) w = Xᵀy with a tiny ridge term λ = 10⁻⁶ for stability. Inputs are scaled to [-1, 1] and a Chebyshev basis is used so the matrix stays well-conditioned.
- Training error only falls as
dgrows (nested models), reaching zero onced + 1 ≥ n. Test (generalisation) error is U-shaped. - The bias-variance trade-off: low degree means high bias (underfitting); high degree means high variance (overfitting), the curve changes a lot with each new sample of noise.
- Errors shown are RMSE. Picking the level from the test dots is a form of validation; a real project also keeps a final untouched test set, or uses cross-validation.
- Remedies: more data, regularisation (larger λ), simpler models, early stopping.
Further reading: An Introduction to Statistical Learning, ch. 2 and 5 · Hastie et al., Elements of Statistical Learning, ch. 7.