Spot the Motion
Two pictures from a video, a moment apart. Can a computer work out what moved, which way, and how far? It plays a giant game of spot-the-difference, one little square at a time.
I cut picture A into little squares called blocks. For each block, I hunt around in picture B for the spot that looks the same. Then I draw an arrow: "this bit moved over there!" Tap any block on the big picture to watch me hunt.
- Look at picture A and picture B. The square slides, the ball rolls, the windmill spins.
- The big picture shows my arrows: green = right, red = wrong.
- Tap a block to see my hunt, then try the challenges below.
Picture A first
Picture B a moment later
Robo's motion arrows
Each arrow starts in the middle of a block and points to where that block went. The yellow box is the block you picked.
Challenges
What just happened?
- I cut picture A into blocks and, for each one, slid it around picture B looking for the best match. The best spot gives the arrow.
- "Best match" means the smallest total difference in brightness, dot by dot. The match map shows that difference for every move I tried.
- If something moves further than my search distance, I simply cannot find it. Faster video needs a bigger search.
- A block with no pattern (the blank wall) looks the same everywhere, so every move is equally good and I just guess. Patterns and corners are what make motion visible.
- Speckle noise confuses small blocks. Bigger blocks average more dots, so the noise cancels out, but they blur the edges between different moving things.
How it works
Cut into blocks
Picture A is chopped into a grid of little squares, like a chocolate bar.
Hunt
Each block is slid to every spot nearby in picture B: left, right, up, down.
Score
At each spot, add up how different every dot looks. Small total = good match.
Draw the arrow
The best spot wins. The arrow goes from the block to that spot.
Check
Because this video is made up, we know the true motion and can mark each arrow right or wrong.
For grown-ups: what's really going on
This is block-matching optical flow, the workhorse of video codecs (MPEG/H.264 motion estimation) and the simplest way to estimate a dense motion field.
- Brightness constancy: we assume a point keeps its brightness as it moves,
IA(x) ≈ IB(x + d). Lighting changes, shadows, noise and blur break this assumption. - Matching cost: for each
b×bblock we exhaustively test every integer displacementdwith|dx|, |dy| ≤ rand minimise the SAD (sum of absolute differences) on luminance. SSD (squared) or normalised cross-correlation are common alternatives; SSD is the maximum-likelihood choice under Gaussian noise, SAD is more robust to outliers. A tiny penaltyλ|d|1breaks ties towards zero motion, and a parabola fitted through the minimum and its neighbours gives sub-pixel estimates. - The aperture problem: a textureless patch constrains nothing, and a patch containing only a straight edge constrains only the motion component perpendicular to that edge. The cost surface becomes a flat plateau or a long valley. The structure tensor (second-moment matrix of the image gradient) measures this: you need two large eigenvalues, i.e. a corner or texture.
- Trade-offs: larger blocks average more pixels (noise variance falls like
1/b²) but assume one translation per block, so they fail at motion boundaries, on rotation and on zoom. The search range must exceed the largest displacement; real systems use coarse-to-fine image pyramids instead of huge windows. - Beyond block matching: Lucas–Kanade linearises brightness constancy and solves a 2×2 least-squares system per window; Horn–Schunck adds a global smoothness term. Modern deep methods such as FlowNet, PWC-Net and RAFT learn features, build a full correlation (cost) volume much like the match map here, and refine the flow iteratively with a recurrent update. They are trained on synthetic data with known ground truth (FlyingChairs, Sintel), exactly as this scene provides.
Evaluation uses the end-point error (EPE): the Euclidean distance between the estimated vector and the ground-truth flow averaged over the block. The scene is procedural: each layer has an analytic motion (translation, rotation, zoom), so ground truth is exact. Frames are 128×96, 2×2 supersampled; noise is Gaussian with σ = 0.08.