Neural Networks

Forgetful on Purpose

A big brain with only a few dots to study can learn every single dot by heart, even the ones that are wrong. Then it fails on new dots. Making brain cells nap at random stops the cramming.

Two brains have already practised 200 steps on the same 40 dots. Some dots are fibbing: they have the wrong colour. The left brain never naps, and look how wiggly its map is! The right brain lets some cells nap each step. Press Practise to keep going and watch the chart below.

How to play: press Practise and compare the two maps. Then tick New (test) dots to see which brain really learned the pattern. Move the sliders on the right to change how the napping brain practises.

What each brain thinks step 0

Know-it-all brainno naps
Napping brain40% nap
Know-it-all: practice / new dots
0% / 0%
Napping: practice / new dots
0% / 0%

Filled dots = the 40 practice dots. Hollow dots = 300 new dots the brains never study. The dashed line is the real border that made the dots.

Oops-score (loss) while practising

know-it-all, practice dots know-it-all, new dots napping, practice dots napping, new dots

Challenges

Catch the crammer: get the know-it-all brain to 100% on practice dots while it scores at least 10 points lower on new dots.
Nap time: with a nap chance of 30% or more, practise 300+ steps. The napping brain's new-dot score must be at least as good, and its oops gap (new minus practice) less than half the know-it-all's.
No naps, just small weights: set nap chance to 0% and weight decay to 0.01 or more. After 300+ steps, beat the know-it-all's oops gap by half again.

What just happened?

How it works

  1. We make 40 practice dots and 300 new dots from the same wavy border, then flip about 1 in 8 colours to act as fibbers.
  2. Both brains start from exactly the same random wires: 2 inputs, two layers of 16 cells, 1 output.
  3. Every step, each brain looks at all 40 practice dots, measures its oops-score, and nudges every wire downhill.
  4. For the napping brain, each cell flips a coin first. Nappers send out nothing, and the awake cells shout a little louder to make up for it.
  5. After each step we score both brains on practice dots and new dots with every cell awake, and draw the lines.
For grown-ups: what's really going on

Both models are multilayer perceptrons 2 → 16 → 16 → 1 with tanh hidden units and a sigmoid output, trained full-batch on binary cross-entropy with Adam (lr 0.02). Data: y > 0.35·sin(2.2x) + ε with 12% label noise, 40 training and 300 test points.

  • Dropout (Srivastava et al., 2014) multiplies each hidden activation by a mask m ~ Bernoulli(1−p). This page uses inverted dropout: kept units are scaled by 1/(1−p) during training, so the expected activation is unchanged and inference simply uses all units with no rescaling. A fresh mask is drawn per example per step; the diagram shows the mask for one example.
  • Dropout trains an implicit ensemble of 232 thinned sub-networks with shared weights and discourages co-adaptation; the full network at test time approximates their geometric-mean prediction.
  • Weight decay adds λ·w to each weight's gradient (L2 penalty ½λ‖w‖²; biases excluded). Note that with Adam this is coupled L2, not the decoupled AdamW variant. Smaller weights keep tanh units in their near-linear range, limiting curvature of the decision boundary.
  • The widening gap between training and test loss is the classic signature of overfitting (high variance). Regularisation trades a slightly worse training fit for lower variance. Test loss for the unregularised net blows up because it becomes confidently wrong on noisy regions.

Everything is seeded (mulberry32), so the same dots and starting weights give the same race. Weight decay slider values: 0, 0.001, 0.003, 0.01, 0.03, 0.1.