Find the Best Knobs
A model has settings, like how many neighbours vote. You only get a few tries, so which settings do you test? And never peek at the final exam!
My model has two knobs: k (how many neighbours vote) and a weight (how much closer neighbours count). Press Search to try a few settings.
Score landscape
square = grid trycircle = random trydarker blue = better validation score
The chosen model
Challenges
Search once: try grid and random search.
Big budget: search with 30 or more tries.
Cheat once: pick the winner by test score, and look at the Test and Fresh columns.
Fair race: press Repeat on 30 datasets and read the averages.
What just happened?
- The model is a real k-nearest-neighbours classifier. Each point is labelled by a vote of its k closest training points; the weight knob makes closer neighbours count more.
- Grid search tries evenly spaced settings. Random search tries random ones. With the same budget, here they usually end up close. Random tends to win when only one knob really matters, because it tests many different values of that knob.
- The validation score is used to choose, so it is a little too hopeful. The fresh score is the honest one.
- If you choose by the test score, the test score becomes the hopeful one, and you no longer have a clean exam left.
- The more tries you allow, the more the best-looking one is partly luck.
How it works
- Split the data: train (the model learns from it), validation (you pick settings with it), test (the final exam).
- Choose which settings to try: a grid, or random ones, with the same budget.
- Score every try on validation data.
- Pick the best settings.
- Report the test score once, at the end. Never tune on it.
For grown-ups: what's really going on
These settings are hyperparameters: chosen by you, not learned from data. Here they are k (1 to 25) and an inverse-distance weight power p (0 to 3): each neighbour votes with weight 1 / (d + 0.05)p.
- Grid search puts the budget on a regular lattice; random search (Bergstra and Bengio, 2012) samples independently. Random is better when some knobs matter much less than others, since a grid wastes tries repeating the same values of the important knob. On this tiny landscape both are usually close, which the 30-dataset average shows honestly.
- Choosing the maximum of many noisy scores gives a winner's curse: the selected score is biased upward. That bias is the gap between Valid. and Fresh.
- Data leakage: tuning on the test set makes it part of training. The test score is then optimistic too. Real projects keep a final untouched set, or use cross-validation and nested cross-validation to keep the estimate honest.
- The dataset is two noisy half-moons with 8% of the labels flipped, so the landscape is bumpy. Validation and test have only 50 points each, so single scores are noisy by several percent.
Further reading: Bergstra and Bengio, Random Search for Hyper-Parameter Optimization, JMLR (2012).