Words ↔ Pictures
I can turn a picture into a list of numbers, and a word into a list of numbers too. If the two lists look alike, they match! Press Wake up the model to start.
Inside the model
Each list of numbers drawn as a barcode
Word map: the picture is pulled toward its matches
How the guess changed
Every time Robo looks, a snapshot lands here. Draw stroke by stroke and watch the guess change! Tap a snapshot to look at it again.
How it works
Chop the picture
The picture is cut into 49 little squares, like a puzzle. The model studies every square and how they fit together.
Picture numbers
A "picture brain" squeezes everything it saw into a list of 512 numbers. That list is the picture's secret code.
Word numbers
A "word brain" turns each word into its own list of 512 numbers, in the very same secret code.
Compare
Multiply the two lists number by number and add it all up. Big total = the codes agree = good match.
Winner
The word whose code agrees most with the picture wins. That's how the model "reads" a picture with words.
What just happened?
- A real model turned your picture into 512 numbers and each word into 512 numbers.
- Pictures and words that mean the same thing get look-alike numbers, because the model practised on 400 million pictures with captions.
- Matching is just "how alike are these two number lists?" No one taught it your exact drawing.
- Adding strokes changed the picture's numbers, so the best word changed too.
For grown-ups: what's really going on
This runs OpenAI's CLIP (ViT-B/32, 8-bit quantised, via Transformers.js + ONNX Runtime WebAssembly) entirely in the browser. Model: Xenova/clip-vit-base-patch32.
- Two encoders, one space: a Vision Transformer splits the 224×224 image into 7×7 patches of 32 px and produces an image embedding; a Transformer text encoder produces a text embedding. Both are projected to 512 dimensions and L2-normalised.
- Contrastive training: trained on ~400M (image, caption) pairs so matching pairs have high cosine similarity and mismatched pairs low (InfoNCE loss).
- Zero-shot classification: similarity
s = ⟨u, v⟩for each label prompt, thensoftmax(100 · s). Raw cosines are small (≈0.2–0.3); the learned temperature (logit scale 100) turns small gaps into confident probabilities. Prompt templates such as “a drawing of a …” noticeably change results. - The barcodes are the actual 512-d vectors; the "agreement" strip is the element-wise product whose sum is the cosine similarity.
- The map places words by PCA of their text embeddings; the picture is drawn at the probability-weighted average of word positions. (Projecting image and text together would mostly show CLIP's "modality gap" between images and texts.)
Further reading: CLIP (OpenAI) · CNN Explainer · Transformer Explainer.