Vision-Language Models · Captioning

Picture Talker

Show me a picture and I'll tell you what I see, one word at a time. Each word is a guess about what comes next, just like a chatbot!

Watch it write

The caption so far

Wake up the model, then press “Describe it!”

Robo's guesses for the next word

What's happening inside

1Chop picture into 196 squares
2Picture brain makes a summary
3Writing brain guesses next word
4Pick a word, add it, repeat
Nothing yet. Press “Describe it!”

Every word, step by step

Each card is one step of writing. Tap a card to see what Robo was choosing between at that moment.

How it works

Chop

The picture is cut into 196 tiny squares, like a mosaic.

Look

The picture brain studies every square and how they fit together, then writes a "summary" made of numbers.

Guess one word

The writing brain reads the summary and the words so far, and scores every word it knows for "what comes next?"

Again and again

Pick a word, glue it on, guess again, until it guesses “the end”.

What just happened?

For grown-ups: what's really going on

Model: Xenova/vit-gpt2-image-captioning (8-bit quantised ONNX), run with Transformers.js on WebAssembly. It is a vision encoder–decoder: a ViT-Base encoder (224×224 image → 14×14 = 196 patches of 16 px → 197 tokens × 768-d) and a GPT-2 decoder that attends to those tokens through cross-attention, trained on COCO captions.

  • Step by step: we call generate() once per word with a custom LogitsProcessor that forces the words already chosen and reads the softmax over the 50,257-token vocabulary at the new position. The encoder output is computed once per picture and reused.
  • Decoding: “Safest word” = greedy (argmax). “Creative” = temperature sampling from the top-40 tokens. “You pick” = human-in-the-loop decoding. A no-repeat-3-gram rule stops loops like “a cat with a cat with a cat”.
  • Tokens, not words: GPT-2 uses byte-pair tokens, so some “words” are word pieces.
  • Asking questions (VQA): modern VLMs (LLaVA, Qwen-VL, SmolVLM, Florence-2…) can answer questions about a picture, but the ones that do so reliably are hundreds of MB to several GB, too heavy to promise a smooth in-browser experience here, so this lesson sticks to captioning.

Further reading: Transformer Explainer · original vit-gpt2 model card.