Ask the Picture
Robo chops a picture into 16 puzzle pieces. When you ask a question, it decides which pieces to look at, then answers. It learns all of this right here, from zero.
My brain is brand new, so my answers are wild guesses. Press Teach Robo and watch me practise on thousands of made-up pictures!
The picture
Tap a square to put a shape there. Tap it again to take it away.
Each piece is just 5 × 5 coloured dots. That's all the clues Robo gets.
A quiz of 200 pictures Robo never practised on. Guessing would score about 33%.
Challenges
What just happened?
- Robo cut the picture into 16 puzzle pieces and turned each one into a short list of numbers.
- Your question became a list of numbers too: a little note saying what to look for.
- Robo compared the note with every piece. Pieces that matched got a big share of its attention (the glow), and it mixed those pieces together.
- A small answer brain read that mix and picked a word or a number.
- Nobody told it where to look. It discovered looking at the right piece helps, just by practising and being corrected.
How it works
Chop
Cut the picture into 16 pieces. Each piece is 5 × 5 coloured dots.
Describe
Turn every piece into 24 numbers, like a tiny description: "reddish, roundish".
Look
The question's numbers are compared with each piece. Good matches get more attention, and the attention always adds up to 100%.
Answer
Mix the pieces by their attention, then a small brain picks the answer: a colour, yes / no, or a number.
For grown-ups: what's really going on
This is a miniature visual question answering (VQA) model, written in plain JavaScript with hand-derived backpropagation and trained with Adam on freshly generated (scene, question, answer) triples. Nothing is downloaded.
- Patches, ViT-style: the 20×20 RGB image is split into a 4×4 grid of 5×5 patches. Each flattened patch (75 values, with a little Gaussian pixel noise during training) passes through a learned patch embedding
e = ReLU(Wpx + b)to 24 dimensions. No positional embedding is used, since no question asks where. - Question as a query: each of the 15 questions has a learned embedding
q(a stand-in for a text encoder). - Cross-attention: keys
ki = Wkei, valuesvi = Wvei, weightsa = softmax(q·ki / √d), pooledc = Σ aivi. The glow is exactlya. This is one head of the cross-attention that VLMs such as Flamingo, BLIP-2 and LLaVA-style models use to let text tokens read image tokens. - Answer head: an MLP on
[c; q](32 ReLU units) gives logits over 9 answers (red, blue, green, yes, no, 0, 1, 2, 3+), trained with softmax cross-entropy, batch 16. - Counting with softmax: attention weights sum to 1, so counting is indirect: the share of attention landing on matching patches versus background grows with the count, and the MLP decodes that. This is why counts are the hardest questions here, as in real VQA benchmarks.
- Generalisation: the quiz is a fixed held-out set of 200 generated examples. Training scenes hold 1 to 5 shapes, so crowded scenes are out of distribution and errors rise, a small taste of distribution shift.
Further reading: ViT (Dosovitskiy et al.) · VQA (Antol et al.) · CLEVR, the synthetic-shapes benchmark this lesson is modelled on.