Video generation

Spot the Motion

Two pictures from a video, a moment apart. Can a computer work out what moved, which way, and how far? It plays a giant game of spot-the-difference, one little square at a time.

I cut picture A into little squares called blocks. For each block, I hunt around in picture B for the spot that looks the same. Then I draw an arrow: "this bit moved over there!" Tap any block on the big picture to watch me hunt.

  1. Look at picture A and picture B. The square slides, the ball rolls, the windmill spins.
  2. The big picture shows my arrows: green = right, red = wrong.
  3. Tap a block to see my hunt, then try the challenges below.

Picture A first

Picture B a moment later

Robo's motion arrows

Each arrow starts in the middle of a block and points to where that block went. The yellow box is the block you picked.

Colour = which way it moved (see the wheel). Pale = barely moved, strong = moved a lot.

Challenges

Catch the speedy square: set the speed to 2× or more, then change my tools until the square's arrows are right (miss under 1 dot). Try this: set the search distance back to 3 and watch the square's arrows go red. I just can't look far enough!
The blank wall puzzle: turn on the blank wall, then tap a block right in the middle of it. Look at the match map. Try this: tap a block on the wall's top edge. The edge is a straight line, so sliding along it still looks the same. Grown-ups call this the aperture problem.
Beat the speckles: at 1× speed with only the noisy camera turned on, get the average miss under 1 dot. Try this: switch to Colours. With tiny blocks the speckles make a messy rainbow; with big blocks the colours calm down.

What just happened?

How it works

Cut into blocks

Picture A is chopped into a grid of little squares, like a chocolate bar.

Hunt

Each block is slid to every spot nearby in picture B: left, right, up, down.

Score

At each spot, add up how different every dot looks. Small total = good match.

Draw the arrow

The best spot wins. The arrow goes from the block to that spot.

Check

Because this video is made up, we know the true motion and can mark each arrow right or wrong.

For grown-ups: what's really going on

This is block-matching optical flow, the workhorse of video codecs (MPEG/H.264 motion estimation) and the simplest way to estimate a dense motion field.

  • Brightness constancy: we assume a point keeps its brightness as it moves, IA(x) ≈ IB(x + d). Lighting changes, shadows, noise and blur break this assumption.
  • Matching cost: for each b×b block we exhaustively test every integer displacement d with |dx|, |dy| ≤ r and minimise the SAD (sum of absolute differences) on luminance. SSD (squared) or normalised cross-correlation are common alternatives; SSD is the maximum-likelihood choice under Gaussian noise, SAD is more robust to outliers. A tiny penalty λ|d|1 breaks ties towards zero motion, and a parabola fitted through the minimum and its neighbours gives sub-pixel estimates.
  • The aperture problem: a textureless patch constrains nothing, and a patch containing only a straight edge constrains only the motion component perpendicular to that edge. The cost surface becomes a flat plateau or a long valley. The structure tensor (second-moment matrix of the image gradient) measures this: you need two large eigenvalues, i.e. a corner or texture.
  • Trade-offs: larger blocks average more pixels (noise variance falls like 1/b²) but assume one translation per block, so they fail at motion boundaries, on rotation and on zoom. The search range must exceed the largest displacement; real systems use coarse-to-fine image pyramids instead of huge windows.
  • Beyond block matching: Lucas–Kanade linearises brightness constancy and solves a 2×2 least-squares system per window; Horn–Schunck adds a global smoothness term. Modern deep methods such as FlowNet, PWC-Net and RAFT learn features, build a full correlation (cost) volume much like the match map here, and refine the flow iteratively with a recurrent update. They are trained on synthetic data with known ground truth (FlyingChairs, Sintel), exactly as this scene provides.

Evaluation uses the end-point error (EPE): the Euclidean distance between the estimated vector and the ground-truth flow averaged over the block. The scene is procedural: each layer has an analytic motion (translation, rotation, zoom), so ground truth is exact. Frames are 128×96, 2×2 supersampled; noise is Gaussian with σ = 0.08.