The Next-Frame Dreamer
A video is just pictures shown one after another. If a computer can guess the next picture, it can make up a whole video!
I want to learn how a bouncing ball moves. Press Start learning and I'll watch lots of tiny videos. Right now my dreams are just fuzz!
Real world
Robo's dream live brain
Both start from the same two pictures. After that, the dream only uses Robo's own guesses. Drag on the real world to throw the ball your way.
How wrong the guesses are (lower is better)
Checkpoints: tap one to watch that old brain dream
How it works
Watch
Robo watches thousands of tiny videos of a bouncing ball. Each picture is only 24 × 24 dots.
Guess the next picture
It looks at two pictures in a row and guesses where every dot will be in the next one.
Check
It compares its guess with the real next picture. Red dots on the oops map are mistakes.
Tweak
It nudges its brain's numbers so the next guess is a tiny bit less wrong. Over and over.
Dream
Now feed it its own guess as the next picture, then guess again… That's a made-up video!
Mistakes pile up
Small mistakes grow with each new picture, so long dreams get blurry. Big video AIs fight this too.
What just happened?
- At first Robo's brain was random, so its dreams were just fuzz.
- By guessing the next picture and checking, it slowly learned that balls keep moving the same way and bounce off walls.
- Once it could guess one picture ahead, it could chain guesses together and dream a whole video.
- The checkpoints show the same brain getting better: fuzz → blob → bouncing ball.
- Nobody told it the rules of bouncing. It found them by watching.
For grown-ups: what's really going on
This is an autoregressive next-frame predictor, the simplest kind of video generator, trained live in your browser.
- Data: endless synthetic clips of 1–2 anti-aliased balls on a 24×24 grid (optionally with gravity). Each training example is a triple of frames
(xt−1, xt) → xt+1; two input frames are needed so the model can infer velocity. - Model: an MLP
1152 → 96 ReLU → 576 sigmoid, written in plain JavaScript. Because input frames are mostly black, the first layer only touches the lit pixels (a sparse matmul), which is what makes in-browser training fast. - Training: per-pixel binary cross-entropy (lit pixels up-weighted ×3 so the ball isn't ignored), mini-batches of 16, Adam (lr 0.002). “Next-picture guess” is the IoU between the thresholded one-step prediction and the true frame; “dream match” is the average IoU over a 15-frame rollout.
- Dreaming: rollouts feed predictions back as inputs. Since the model was only trained on real frames, its own slightly-blurry outputs are out-of-distribution, so errors compound. This is called exposure bias.
How real video generators extend this: modern systems (e.g. OpenAI's Sora, Google's Veo) first compress video into a smaller latent space with an autoencoder, cut it into spacetime patches (little cubes of space × time), and use a diffusion transformer to denoise all patches of a clip together rather than one frame at a time. That joint denoising keeps long clips consistent. Text conditioning steers what appears. Autoregressive “world models” (like DeepMind's Genie) are much closer to this demo: predict the next frame, given the past and an action.
Further reading: Video generation models as world simulators (OpenAI), and Scalable Diffusion Models with Transformers (Peebles & Xie).