Generative Models

Squeeze and Rebuild

A small neural network squeezes each picture into just a few numbers, then tries to rebuild it. Make the squeeze tighter and watch what gets lost.

These are tiny 8×8 pictures: 64 pixels each. I squash every picture into only 4 numbers and rebuild it. Drag the slider to change how many numbers I may keep!

Original, secret code, rebuilt

These 8 pictures were never used for learning, so the rebuilt row is an honest test.

64 pixels go in, only this many numbers come out of the middle, then the picture is rebuilt from them.

Challenges

Keep plenty: set the bottleneck to 12 or more and train until the error on new pictures is under 0.035.
Squeeze hard: set it to 2 or fewer, train 500 steps, and see the error stay above 0.045 (blurry!).
Try them all: run the sweep and watch error drop as the bottleneck grows.
Find a bargain: reach error under 0.035 with a bottleneck of 8 or fewer numbers.

What just happened?

For grown-ups: what's really going on

This is an autoencoder: a neural network trained to output its own input. Here it is 64 → 24 → k → 24 → 64 with tanh hidden units, a linear bottleneck of k units and a sigmoid output. It is a real network, trained in your browser with backpropagation and the Adam optimiser on mini-batches of 16 pictures.

  • Loss is mean squared error per pixel between input and rebuilt picture. The number in the box is that loss on 48 held-out pictures.
  • The bottleneck forces lossy compression. Fewer latent units means a smaller code and a higher error floor. It is a non-linear cousin of PCA.
  • Error on the practice pictures can fall below the error on new ones: that gap is overfitting.
  • Pictures are procedurally drawn squares, rings, plus signs, slashes, triangles and stripes with random size and position plus a little noise.

Further reading: Hinton and Salakhutdinov, Reducing the dimensionality of data with neural networks (Science, 2006).