Optimisation & Search

Find the Best Knobs

A model has settings, like how many neighbours vote. You only get a few tries, so which settings do you test? And never peek at the final exam!

My model has two knobs: k (how many neighbours vote) and a weight (how much closer neighbours count). Press Search to try a few settings.

Score landscape

square = grid trycircle = random trydarker blue = better validation score

The chosen model

Challenges

Big budget: search with 30 or more tries.
Cheat once: pick the winner by test score, and look at the Test and Fresh columns.
Fair race: press Repeat on 30 datasets and read the averages.

What just happened?

How it works

  1. Split the data: train (the model learns from it), validation (you pick settings with it), test (the final exam).
  2. Choose which settings to try: a grid, or random ones, with the same budget.
  3. Score every try on validation data.
  4. Pick the best settings.
  5. Report the test score once, at the end. Never tune on it.
For grown-ups: what's really going on

These settings are hyperparameters: chosen by you, not learned from data. Here they are k (1 to 25) and an inverse-distance weight power p (0 to 3): each neighbour votes with weight 1 / (d + 0.05)p.

  • Grid search puts the budget on a regular lattice; random search (Bergstra and Bengio, 2012) samples independently. Random is better when some knobs matter much less than others, since a grid wastes tries repeating the same values of the important knob. On this tiny landscape both are usually close, which the 30-dataset average shows honestly.
  • Choosing the maximum of many noisy scores gives a winner's curse: the selected score is biased upward. That bias is the gap between Valid. and Fresh.
  • Data leakage: tuning on the test set makes it part of training. The test score is then optimistic too. Real projects keep a final untouched set, or use cross-validation and nested cross-validation to keep the estimate honest.
  • The dataset is two noisy half-moons with 8% of the labels flipped, so the landscape is bumpy. Validation and test have only 50 points each, so single scores are noisy by several percent.

Further reading: Bergstra and Bengio, Random Search for Hyper-Parameter Optimization, JMLR (2012).