Generative Models

Make New Pictures

Pick random spots on the map and the network draws brand-new pictures. A tidy-up slider decides whether the map is full of holes or a blurry mush.

I will draw pictures from random spots. If the map has holes I draw junk. Slide the tidy-up weight and watch the junk and the blur change!

The map

Faint tiles are what the network draws at each spot. Dots are real pictures. The dashed circle is where random spots usually land. White rings are your new pictures.

0 = no rule about the map. Higher = it must keep the dots packed in the circle, at the cost of detail.

Challenges

No rules: set the weight to 0 and let it train until Spread is over 20. The dots sprawl far outside the circle, leaving holes.
Sweet spot: find a weight where junk is 30% or less and the pictures stay sharp (0.7 or more).
Too much tidying: set the weight to 3 and see sharpness drop to 0.65 or less.
Experimenter: train with 4 different weights.

What just happened?

For grown-ups: what's really going on

A variational autoencoder (VAE) has an encoder that outputs a mean μ and log-variance log σ² for a 2-D Gaussian, and a decoder (2 → 32 → 64). Training minimises ‖x − x̂‖² + β · KL.

  • Reparameterisation trick: z = μ + σ · ε with ε ~ N(0, 1), so gradients flow through the sampling step into the encoder. This is implemented by hand here.
  • KL divergence to the prior N(0, I): −½ Σ (1 + log σ² − μ² − σ²). It is the regulariser. With β = 0 it is a plain autoencoder: the codes spread far outside the prior, so sampling from N(0, I) often lands in untrained gaps (junk).
  • Large β causes posterior collapse in the limit: the encoder ignores the input, and the decoder outputs the average picture.
  • This is a β-VAE view: the weight is on the KL term relative to the squared-error reconstruction term (a Gaussian likelihood with a fixed variance).
  • "Junk" = nearest training picture has per-pixel mean squared difference above 0.05. "Sharp" = mean of |pixel − 0.5| × 2: 1 for pure black and white, 0 for mid grey.

Further reading: Kingma and Welling, Auto-Encoding Variational Bayes (2014); Higgins et al., β-VAE (2017).