Forgetful on Purpose
A big brain with only a few dots to study can learn every single dot by heart, even the ones that are wrong. Then it fails on new dots. Making brain cells nap at random stops the cramming.
Two brains have already practised 200 steps on the same 40 dots. Some dots are fibbing: they have the wrong colour. The left brain never naps, and look how wiggly its map is! The right brain lets some cells nap each step. Press Practise to keep going and watch the chart below.
How to play: press Practise and compare the two maps. Then tick New (test) dots to see which brain really learned the pattern. Move the sliders on the right to change how the napping brain practises.
What each brain thinks step 0
Filled dots = the 40 practice dots. Hollow dots = 300 new dots the brains never study. The dashed line is the real border that made the dots.
Oops-score (loss) while practising
Challenges
What just happened?
- The know-it-all brain has lots of cells and only 40 dots, so it can bend its border around every dot, even the fibbing ones. Practice score: 100%. New dots: much worse.
- On the chart, its practice oops-line drops to almost zero while its new-dot line turns round and climbs. When those two lines pull apart, the brain is memorising, not learning.
- In the napping brain, a different random group of cells is asleep every step. No cell can count on its neighbours, so no single cell can memorise one odd dot. The border stays smooth and close to the real one.
- Weight decay does a similar job in a different way: it keeps every wire small, and small wires can only draw gentle curves.
- When the brain is used for real, nobody naps. Every cell is awake, and the answer is like an average of all the smaller brains it practised as.
How it works
- We make 40 practice dots and 300 new dots from the same wavy border, then flip about 1 in 8 colours to act as fibbers.
- Both brains start from exactly the same random wires: 2 inputs, two layers of 16 cells, 1 output.
- Every step, each brain looks at all 40 practice dots, measures its oops-score, and nudges every wire downhill.
- For the napping brain, each cell flips a coin first. Nappers send out nothing, and the awake cells shout a little louder to make up for it.
- After each step we score both brains on practice dots and new dots with every cell awake, and draw the lines.
For grown-ups: what's really going on
Both models are multilayer perceptrons 2 → 16 → 16 → 1 with tanh hidden units and a sigmoid output, trained full-batch on binary cross-entropy with Adam (lr 0.02). Data: y > 0.35·sin(2.2x) + ε with 12% label noise, 40 training and 300 test points.
- Dropout (Srivastava et al., 2014) multiplies each hidden activation by a mask
m ~ Bernoulli(1−p). This page uses inverted dropout: kept units are scaled by1/(1−p)during training, so the expected activation is unchanged and inference simply uses all units with no rescaling. A fresh mask is drawn per example per step; the diagram shows the mask for one example. - Dropout trains an implicit ensemble of
232thinned sub-networks with shared weights and discourages co-adaptation; the full network at test time approximates their geometric-mean prediction. - Weight decay adds
λ·wto each weight's gradient (L2 penalty½λ‖w‖²; biases excluded). Note that with Adam this is coupled L2, not the decoupled AdamW variant. Smaller weights keep tanh units in their near-linear range, limiting curvature of the decision boundary. - The widening gap between training and test loss is the classic signature of overfitting (high variance). Regularisation trades a slightly worse training fit for lower variance. Test loss for the unregularised net blows up because it becomes confidently wrong on noisy regions.
Everything is seeded (mulberry32), so the same dots and starting weights give the same race. Weight decay slider values: 0, 0.001, 0.003, 0.01, 0.03, 0.1.