Computer Vision

Pattern Scanners

A tiny window slides over a picture and gives every spot a score for how much it looks like a pattern. The scanner's nine numbers decide the pattern, and a robot can learn them all by itself.

I scan with a little 3 by 3 window. Each spot gets multiply, then add. Draw on the picture, then drag the scanner's numbers up and down to light up the target map!

Part 1 · Play with the scanner

The picture. Tap or drag to paint. Orange box = where the scanner is now.
Score map (yours)
Target map (the answer)

Part 2 · Watch it learn

Mistake size over time

Steps
0
Mistake
–

Challenges

Each win puts a flag on the chart above. The last two use the buttons below.

How it works

Slide a window

A 3 by 3 window visits every spot of the picture.

Multiply and add

Pixel times scanner number, nine times, then add them up. That is the spot's score.

Compare to the answer

The robot checks how far its score map is from the right one.

Nudge the nine numbers

Small nudges, again and again, until the maps match.

What just happened?

For grown-ups: what's really going on

This is a single 2D convolution layer with one kernel (filter). Like most deep-learning libraries, it actually computes cross-correlation: y[i,j] = Σₐ Σ_b x[i+a, j+b]·w[a,b], without flipping the kernel. The output grid is a feature map. Here there is no padding (a "valid" convolution, so 12x12 becomes 10x10) and the stride is 1.

  • Training minimises MSE = mean (y − t)² over every position of several random pictures. The gradient is ∂MSE/∂w[a,b] = mean 2(y−t)·x[i+a, j+b], and the update is plain gradient descent: w ← w − η∇. The loss is quadratic in w, so a small enough η gives a steady decrease (the chart is on a log scale).
  • The hidden kernels are Sobel-style edge detectors and a corner detector. A real CNN is not told the answer; it learns kernels that reduce a task loss, and the kernels often end up looking like edges.
  • Max pooling (2x2, stride 2) keeps the strongest response in each block, shrinking the map and making it tolerant of small shifts.
  • Weight sharing gives translation equivariance: shift the input and the feature map shifts with it. Pooling and a final classifier turn this into approximate invariance.
  • A CNN stacks many such layers (with many kernels per layer, a nonlinearity such as ReLU, and pooling). Early layers find edges, later layers combine them into shapes and objects. This lesson is one kernel from one layer.

Further reading: Deep Learning (Goodfellow et al.), ch. 9 · CS231n notes on convolutional networks.