Picture Talker
Show me a picture and I'll tell you what I see, one word at a time. Each word is a guess about what comes next, just like a chatbot!
Watch it write
The caption so far
Robo's guesses for the next word
What's happening inside
Every word, step by step
Each card is one step of writing. Tap a card to see what Robo was choosing between at that moment.
How it works
Chop
The picture is cut into 196 tiny squares, like a mosaic.
Look
The picture brain studies every square and how they fit together, then writes a "summary" made of numbers.
Guess one word
The writing brain reads the summary and the words so far, and scores every word it knows for "what comes next?"
Again and again
Pick a word, glue it on, guess again, until it guesses “the end”.
What just happened?
- The model never "sees" like you do: it turned your picture into lots of numbers first.
- Then it wrote the caption one word at a time, each time choosing from a list of guesses with chances.
- “Safest word” always takes the top guess. “Creative” rolls the dice. When you pick, it keeps writing from your word.
- This is how chatbots that can see pictures work too, just much, much bigger.
For grown-ups: what's really going on
Model: Xenova/vit-gpt2-image-captioning (8-bit quantised ONNX), run with Transformers.js on WebAssembly. It is a vision encoder–decoder: a ViT-Base encoder (224×224 image → 14×14 = 196 patches of 16 px → 197 tokens × 768-d) and a GPT-2 decoder that attends to those tokens through cross-attention, trained on COCO captions.
- Step by step: we call
generate()once per word with a customLogitsProcessorthat forces the words already chosen and reads the softmax over the 50,257-token vocabulary at the new position. The encoder output is computed once per picture and reused. - Decoding: “Safest word” = greedy (argmax). “Creative” = temperature sampling from the top-40 tokens. “You pick” = human-in-the-loop decoding. A no-repeat-3-gram rule stops loops like “a cat with a cat with a cat”.
- Tokens, not words: GPT-2 uses byte-pair tokens, so some “words” are word pieces.
- Asking questions (VQA): modern VLMs (LLaVA, Qwen-VL, SmolVLM, Florence-2…) can answer questions about a picture, but the ones that do so reliably are hundreds of MB to several GB, too heavy to promise a smooth in-browser experience here, so this lesson sticks to captioning.
Further reading: Transformer Explainer · original vit-gpt2 model card.