Steer the Drawing
Picture-making AIs start from fuzzy static and clean it up. You also tell them what to draw. A secret steering knob decides how hard they listen.
These are the four shapes I'll practise: a ring, a heart, two moons and a star. Press Start learning so I can learn to make them out of fog!
Robo's drawing board
practice shapesFour knobs at once
Same brain, same starting fog, same shape. Only the steering knob changes.
Challenges
What just happened?
- Robo learned one job: look at a fuzzy dot and guess which way the fuzz pushed it.
- Most of the time we told it the shape name. Sometimes we didn't, so it also learned to clean up towards any of its shapes.
- When drawing, Robo makes two guesses for every dot: "with your shape" and "any shape". The knob stretches the difference between them.
- Knob 0: it ignores you and you get a mix. Knob 1 to 3: crisp, tidy shape. Huge knob: it pushes so hard the drawing breaks.
- Big picture-making AIs use this exact trick so your words steer the picture.
How it works
Shake
Take dots from a shape and shake them with random fuzz. Remember the fuzz!
Guess
Show Robo the shaken dot, how fuzzy it is, and usually the shape name. It guesses the fuzz.
Two guesses
When drawing, Robo guesses the fuzz twice: knowing the shape, and not knowing it.
Steer
Start at "not knowing", then move towards "knowing" by the knob amount, and clean a little. Repeat 50 times.
For grown-ups: what's really going on
This is a 2-D DDPM (Ho et al. 2020) with classifier-free guidance (Ho & Salimans 2022), trained live in plain JavaScript.
- ε-prediction: an MLP (3 ReLU layers of 64) predicts the noise
ε̂θ(xt, t, c)from Fourier features of the 2-D point, a sinusoidal timestep embedding and a one-hot class. Loss:‖ε − ε̂θ(√ᾱt x0 + √(1−ᾱt) ε, t, c)‖², linear β schedule 1e-4 → 0.08, T = 100, Adam, batch 64. - Label dropout: with probability 0.2 the class is replaced by a null token (all zeros), so the same network learns the unconditional model
εu = ε̂(xt, t, ∅)and the conditionalεc = ε̂(xt, t, c). - Guidance: at sampling time
ε̃ = εu + w(εc − εu).w = 0is the unconditional model (a mixture of all four shapes),w = 1is plain conditional sampling, andw > 1amounts to sampling from roughlyp(x)·p(c|x)w, which sharpens towards points the implicit classifier is sure about. Too large awovershoots: samples leave the data manifold and lose diversity. (Some papers write(1+w); this lesson'swis the "guidance scale" used by Stable Diffusion, typically 5 to 9 for images.) - Sampler: ancestral DDPM with 50 respaced steps,
xt−1 = (xt − βt/√(1−ᾱt) · ε̃)/√αt + σtz. - Scores: "On the shape" is the share of samples within 0.12 of a real point (a precision proxy); "Fills the shape" is the share of 600 real points with a sample within 0.12 (a recall / coverage proxy). Guidance trades recall for precision, the same fidelity-versus-diversity trade-off seen in FID / IS curves for image models.
Further reading: Classifier-Free Diffusion Guidance · DDPM · Diffusion Explainer (which has a guidance-scale slider for Stable Diffusion).