Video generation

In-Between Frames

Slow motion needs extra pictures between the real ones. The camera never took them, so the computer has to make them up. Some ways look smooth; some make spooky ghosts!

I only have picture A and picture B. The big picture is my made-up picture from exactly halfway between them. I made it by following the motion arrows. Press Play slow-mo, then try the other ways!

  1. Pick a way to make frames on the right.
  2. Drag the time slider from A to B, or press Play slow-mo.
  3. Compare my made-up picture with the real one (middle, top). The bar chart shows who is closest.

Picture A

Real t = 0.50

Picture B

Robo's made-up picture

t = 0.50
Time slider: left end is picture A, right end is picture B. Everything in between is made up.

The slow-mo film strip

Grey labels are real pictures, coloured labels are made up. Tap a frame to jump to it.

Challenges

Ghost hunter: choose Fade and put the time slider near the middle. Can you see two see-through squares? Try this: play the Fade at 8× slow-mo. Things don't move at all: one copy fades out while another fades in.
Super slow-mo: with speed 2× or more, choose Follow arrows and play it all the way through at 8× slow-mo. Try this: play Repeat at 8× too. It just sits still and then jumps: that's the jerky look of cheap slow motion.
Find the tear: choose Follow arrows with Perfect arrows, switch to the Oops map, and tap a spot that is still red. Try this: drag the time slider. The red bits sit just behind and in front of moving things, where the background was hidden in one of the pictures.

What just happened?

How it works

Two real pictures

We have picture A and picture B, a moment apart.

Find the arrows

Block matching finds which way each bit moved, just like Spot the Motion.

Pick a moment

Halfway? Then each bit should be halfway along its arrow.

Fetch from both

For every dot, look back along the arrow into A and forward into B, and fetch the colour from both.

Mix

Mix the two colours, using more of the nearer picture. Repeat for every made-up frame.

For grown-ups: what's really going on

This is video frame interpolation (frame-rate up-conversion), used for slow motion, 24→60 fps TV "motion smoothing" and video codecs (B-frames).

  • Frame repetition (sample-and-hold) is It = I0 for t ≤ ½, else I1: judder.
  • Cross-fade is It = (1−t)I0 + tI1. Its error at moving edges is a double exposure ("ghosting"). It minimises MSE only when there is no motion.
  • Motion-compensated interpolation uses backward warping: for each output pixel p it finds a vector v passing through p at time t, then samples (bilinearly) I0(p − t·v) and I1(p + (1−t)·v) and blends them with weights 1−t and t. Since the flow is estimated on frame A's grid, not at time t, we pick v per pixel from nearby flow vectors by minimising a symmetric (bilateral) 3×3 SAD between the two fetches, a classic TV-chip approach. Robo's arrows are 8×8 block matches (search ±12, sub-pixel), refined per pixel.
  • Occlusion: background uncovered between A and B exists in only one frame, so blending both fetches mixes foreground into it (the "tear" or halo). The fix is visibility-aware blending: estimate occlusion masks and fetch only from the frame where the pixel is visible. This demo deliberately leaves that out so you can see the artefact; the "Perfect arrows" mode shows it is not a flow-accuracy problem.
  • Deep interpolators: Super SloMo approximates intermediate flows from bidirectional flow and learns soft visibility maps; DAIN adds depth to decide which object is in front when vectors collide; RIFE directly predicts intermediate flows with a fast coarse-to-fine network plus a fusion mask; Google's FILM uses multi-scale feature extraction with shared weights to handle large motion and synthesises the frame from warped features rather than raw pixels, reducing tears.

Error is root-mean-square error over RGB in 0..1, shown as a percentage, against the exact frame rendered at time t from the procedural scene. The bar chart always uses t = ½, where interpolation is hardest.