Vision-Language Models · Visual Question Answering

Ask the Picture

Robo chops a picture into 16 puzzle pieces. When you ask a question, it decides which pieces to look at, then answers. It learns all of this right here, from zero.

My brain is brand new, so my answers are wild guesses. Press Teach Robo and watch me practise on thousands of made-up pictures!

The picture

Tap a square to put a shape there. Tap it again to take it away.

What Robo really sees

Each piece is just 5 × 5 coloured dots. That's all the clues Robo gets.

Quiz score while practising

A quiz of 200 pictures Robo never practised on. Guessing would score about 33%.

Challenges

Good student: teach Robo until its quiz score reaches 85%. Try this: press New brain and teach again. Every brain learns a little differently, but they all get there.
Spotlight: ask "What colour is the …?" and get a right answer where the shape you asked about glows brightest. Try this: drag your shape somewhere else (tap to remove, tap to place). The glow should follow it.
Catch it out: once Robo scores 85%, build a picture where it gives a wrong answer. Hint: Robo only ever practised with 1 to 5 shapes. Try this: count how many shapes your tricky picture has. Models often go wrong on things unlike their practice.

What just happened?

How it works

Chop

Cut the picture into 16 pieces. Each piece is 5 × 5 coloured dots.

Describe

Turn every piece into 24 numbers, like a tiny description: "reddish, roundish".

Look

The question's numbers are compared with each piece. Good matches get more attention, and the attention always adds up to 100%.

Answer

Mix the pieces by their attention, then a small brain picks the answer: a colour, yes / no, or a number.

For grown-ups: what's really going on

This is a miniature visual question answering (VQA) model, written in plain JavaScript with hand-derived backpropagation and trained with Adam on freshly generated (scene, question, answer) triples. Nothing is downloaded.

  • Patches, ViT-style: the 20×20 RGB image is split into a 4×4 grid of 5×5 patches. Each flattened patch (75 values, with a little Gaussian pixel noise during training) passes through a learned patch embedding e = ReLU(Wpx + b) to 24 dimensions. No positional embedding is used, since no question asks where.
  • Question as a query: each of the 15 questions has a learned embedding q (a stand-in for a text encoder).
  • Cross-attention: keys ki = Wkei, values vi = Wvei, weights a = softmax(q·ki / √d), pooled c = Σ aivi. The glow is exactly a. This is one head of the cross-attention that VLMs such as Flamingo, BLIP-2 and LLaVA-style models use to let text tokens read image tokens.
  • Answer head: an MLP on [c; q] (32 ReLU units) gives logits over 9 answers (red, blue, green, yes, no, 0, 1, 2, 3+), trained with softmax cross-entropy, batch 16.
  • Counting with softmax: attention weights sum to 1, so counting is indirect: the share of attention landing on matching patches versus background grows with the count, and the MLP decodes that. This is why counts are the hardest questions here, as in real VQA benchmarks.
  • Generalisation: the quiz is a fixed held-out set of 200 generated examples. Training scenes hold 1 to 5 shapes, so crowded scenes are out of distribution and errors rise, a small taste of distribution shift.

Further reading: ViT (Dosovitskiy et al.) · VQA (Antol et al.) · CLEVR, the synthetic-shapes benchmark this lesson is modelled on.