Train vs Test
Here is a pile of 100 cards. Keep some to study, hide the rest for an exam. Then see who really learned and who only memorised.
Drag cards to study with. Solid dots are my study cards. Hollow dots are secret exam cards. The colours are my guess for every spot.
Snapshots while studying
Press Robot, study!. At each card count below I take a photo. Tap one to go back to it.
Challenges
What just happened?
- The model only learned from the study cards, then was graded on exam cards it had never seen.
- Looking things up from study cards gives 100% on the study cards. That score tells you very little.
- The exam score is the honest one. With few study cards it is low; with more, it climbs and the gap shrinks.
- Peeking at the exam cards makes both scores look great, but it is cheating. It no longer tests anything new.
- A different random split gives a different exam score, so one number is a bit lucky or unlucky.
For grown-ups: what's really going on
The robot is a 1-nearest-neighbour classifier (k-NN with k = 1): it predicts the label of the closest training point. It has zero bias towards a smooth boundary, so it fits the training set perfectly (training accuracy 100% unless two identical points disagree). That is memorisation, and it is an overfit model by construction on noisy data.
- The data are the two moons with Gaussian noise. The pool of 100 cards is shuffled and cut into a train and a test (hold-out) set; the slider sets the split.
- Generalisation error is estimated on the test set. As training data grows, test accuracy rises toward the Bayes-rate limit set by the noise (the learning curve).
- Data leakage: letting test cards into training inflates the test score to 100% and says nothing about new data. We also show a score on 300 fresh cards that never touched training.
- The test score of one split is a noisy estimate (small test sets have high variance). Remedies: repeated splits, k-fold cross-validation, and keeping one final untouched test set.
Further reading: An Introduction to Statistical Learning, ch. 5 · scikit-learn user guide, "Cross-validation: evaluating estimator performance".