Supervised Learning

Train vs Test

Here is a pile of 100 cards. Keep some to study, hide the rest for an exam. Then see who really learned and who only memorised.

Drag cards to study with. Solid dots are my study cards. Hollow dots are secret exam cards. The colours are my guess for every spot.

blue cards orange cards exam (hidden)

Snapshots while studying

Press Robot, study!. At each card count below I take a photo. Tap one to go back to it.

Challenges

Peek at the exam: press the button and watch both scores hit 100%. Is that real?
Noisier cards: set noise to 0.55 or more with 20+ study cards. The exam score stays much lower.
Unlucky split: deal 5 different splits with the same settings. Does the exam score stay the same?

What just happened?

For grown-ups: what's really going on

The robot is a 1-nearest-neighbour classifier (k-NN with k = 1): it predicts the label of the closest training point. It has zero bias towards a smooth boundary, so it fits the training set perfectly (training accuracy 100% unless two identical points disagree). That is memorisation, and it is an overfit model by construction on noisy data.

  • The data are the two moons with Gaussian noise. The pool of 100 cards is shuffled and cut into a train and a test (hold-out) set; the slider sets the split.
  • Generalisation error is estimated on the test set. As training data grows, test accuracy rises toward the Bayes-rate limit set by the noise (the learning curve).
  • Data leakage: letting test cards into training inflates the test score to 100% and says nothing about new data. We also show a score on 300 fresh cards that never touched training.
  • The test score of one split is a noisy estimate (small test sets have high variance). Remedies: repeated splits, k-fold cross-validation, and keeping one final untouched test set.

Further reading: An Introduction to Statistical Learning, ch. 5 · scikit-learn user guide, "Cross-validation: evaluating estimator performance".