Image Generation · Guidance

Steer the Drawing

Picture-making AIs start from fuzzy static and clean it up. You also tell them what to draw. A secret steering knob decides how hard they listen.

These are the four shapes I'll practise: a ring, a heart, two moons and a star. Press Start learning so I can learn to make them out of fog!

Robo's drawing board

practice shapes
Slide the fog level to rewind: 100% is pure static, 0% is the finished drawing.

Four knobs at once

Same brain, same starting fog, same shape. Only the steering knob changes.

Challenges

Nice and crisp: with the knob at 1 or more, draw any shape with 85% or more of the dots on the shape. Try this: draw each of the four shapes. Which one is hardest for Robo?
Are you listening? Draw a shape with the knob at 0, then the same shape at 2 or more, and see "On the shape" jump by 20 points. Try this: at knob 0, look closely. Can you spot bits of the other shapes mixed in?
Too much! Turn the knob to 6 or more and get "Fills the shape" down to 65% or less. Try this: watch the trails at knob 8. The dots get shoved too hard and bunch up or fly off. Real AIs use a knob of about 3 to 8 on pictures, but too much makes colours harsh.

What just happened?

How it works

Shake

Take dots from a shape and shake them with random fuzz. Remember the fuzz!

Guess

Show Robo the shaken dot, how fuzzy it is, and usually the shape name. It guesses the fuzz.

Two guesses

When drawing, Robo guesses the fuzz twice: knowing the shape, and not knowing it.

Steer

Start at "not knowing", then move towards "knowing" by the knob amount, and clean a little. Repeat 50 times.

For grown-ups: what's really going on

This is a 2-D DDPM (Ho et al. 2020) with classifier-free guidance (Ho & Salimans 2022), trained live in plain JavaScript.

  • ε-prediction: an MLP (3 ReLU layers of 64) predicts the noise ε̂θ(xt, t, c) from Fourier features of the 2-D point, a sinusoidal timestep embedding and a one-hot class. Loss: ‖ε − ε̂θ(√ᾱt x0 + √(1−ᾱt) ε, t, c)‖², linear β schedule 1e-4 → 0.08, T = 100, Adam, batch 64.
  • Label dropout: with probability 0.2 the class is replaced by a null token (all zeros), so the same network learns the unconditional model εu = ε̂(xt, t, ∅) and the conditional εc = ε̂(xt, t, c).
  • Guidance: at sampling time ε̃ = εu + w(εc − εu). w = 0 is the unconditional model (a mixture of all four shapes), w = 1 is plain conditional sampling, and w > 1 amounts to sampling from roughly p(x)·p(c|x)w, which sharpens towards points the implicit classifier is sure about. Too large a w overshoots: samples leave the data manifold and lose diversity. (Some papers write (1+w); this lesson's w is the "guidance scale" used by Stable Diffusion, typically 5 to 9 for images.)
  • Sampler: ancestral DDPM with 50 respaced steps, xt−1 = (xt − βt/√(1−ᾱt) · ε̃)/√αt + σtz.
  • Scores: "On the shape" is the share of samples within 0.12 of a real point (a precision proxy); "Fills the shape" is the share of 600 real points with a sample within 0.12 (a recall / coverage proxy). Guidance trades recall for precision, the same fidelity-versus-diversity trade-off seen in FID / IS curves for image models.

Further reading: Classifier-Free Diffusion Guidance · DDPM · Diffusion Explainer (which has a guidance-scale slider for Stable Diffusion).