Squeeze and Rebuild
A small neural network squeezes each picture into just a few numbers, then tries to rebuild it. Make the squeeze tighter and watch what gets lost.
These are tiny 8×8 pictures: 64 pixels each. I squash every picture into only 4 numbers and rebuild it. Drag the slider to change how many numbers I may keep!
Original, secret code, rebuilt
These 8 pictures were never used for learning, so the rebuilt row is an honest test.
Challenges
What just happened?
- The middle of the network is a bottleneck: only a few numbers can get through, so it has to keep the most important parts of each picture.
- The first half squeezes (encodes), the second half rebuilds (decodes). Nobody told it what the shapes are. It only learns to make its output match its input.
- With a tight squeeze it forgets detail, so the pictures come back blurry and in-between. With a wide one it can copy almost perfectly.
- The orange line is measured on pictures it never practised on, so it is an honest score.
For grown-ups: what's really going on
This is an autoencoder: a neural network trained to output its own input. Here it is 64 → 24 → k → 24 → 64 with tanh hidden units, a linear bottleneck of k units and a sigmoid output. It is a real network, trained in your browser with backpropagation and the Adam optimiser on mini-batches of 16 pictures.
- Loss is mean squared error per pixel between input and rebuilt picture. The number in the box is that loss on 48 held-out pictures.
- The bottleneck forces lossy compression. Fewer latent units means a smaller code and a higher error floor. It is a non-linear cousin of PCA.
- Error on the practice pictures can fall below the error on new ones: that gap is overfitting.
- Pictures are procedurally drawn squares, rings, plus signs, slashes, triangles and stripes with random size and position plus a little noise.
Further reading: Hinton and Salakhutdinov, Reducing the dimensionality of data with neural networks (Science, 2006).