In-Between Frames
Slow motion needs extra pictures between the real ones. The camera never took them, so the computer has to make them up. Some ways look smooth; some make spooky ghosts!
I only have picture A and picture B. The big picture is my made-up picture from exactly halfway between them. I made it by following the motion arrows. Press Play slow-mo, then try the other ways!
- Pick a way to make frames on the right.
- Drag the time slider from A to B, or press Play slow-mo.
- Compare my made-up picture with the real one (middle, top). The bar chart shows who is closest.
Picture A
Real t = 0.50
Picture B
Robo's made-up picture
The slow-mo film strip
Grey labels are real pictures, coloured labels are made up. Tap a frame to jump to it.
Challenges
What just happened?
- Repeat just shows picture A, then picture B. No new information, so the slow-mo looks jerky.
- Fade mixes the two pictures like see-through paper. Moving things show up twice, faded: ghosts!
- Follow arrows first works out where every bit moved (the motion arrows), then puts each bit part of the way along its arrow. Things really slide.
- Even perfect arrows leave small mistakes where something was hidden in one picture, like the background just behind a moving ball. Nobody saw it there, so it has to be guessed.
How it works
Two real pictures
We have picture A and picture B, a moment apart.
Find the arrows
Block matching finds which way each bit moved, just like Spot the Motion.
Pick a moment
Halfway? Then each bit should be halfway along its arrow.
Fetch from both
For every dot, look back along the arrow into A and forward into B, and fetch the colour from both.
Mix
Mix the two colours, using more of the nearer picture. Repeat for every made-up frame.
For grown-ups: what's really going on
This is video frame interpolation (frame-rate up-conversion), used for slow motion, 24→60 fps TV "motion smoothing" and video codecs (B-frames).
- Frame repetition (sample-and-hold) is
It = I0fort ≤ ½, elseI1: judder. - Cross-fade is
It = (1−t)I0 + tI1. Its error at moving edges is a double exposure ("ghosting"). It minimises MSE only when there is no motion. - Motion-compensated interpolation uses backward warping: for each output pixel
pit finds a vectorvpassing throughpat timet, then samples (bilinearly)I0(p − t·v)andI1(p + (1−t)·v)and blends them with weights1−tandt. Since the flow is estimated on frame A's grid, not at timet, we pickvper pixel from nearby flow vectors by minimising a symmetric (bilateral) 3×3 SAD between the two fetches, a classic TV-chip approach. Robo's arrows are 8×8 block matches (search ±12, sub-pixel), refined per pixel. - Occlusion: background uncovered between A and B exists in only one frame, so blending both fetches mixes foreground into it (the "tear" or halo). The fix is visibility-aware blending: estimate occlusion masks and fetch only from the frame where the pixel is visible. This demo deliberately leaves that out so you can see the artefact; the "Perfect arrows" mode shows it is not a flow-accuracy problem.
- Deep interpolators: Super SloMo approximates intermediate flows from bidirectional flow and learns soft visibility maps; DAIN adds depth to decide which object is in front when vectors collide; RIFE directly predicts intermediate flows with a fast coarse-to-fine network plus a fusion mask; Google's FILM uses multi-scale feature extraction with shared weights to handle large motion and synthesises the frame from warped features rather than raw pixels, reducing tears.
Error is root-mean-square error over RGB in 0..1, shown as a percentage, against the exact frame rendered at time t from the procedural scene. The bar chart always uses t = ½, where interpolation is hardest.