Vision-Language Models · CLIP

Words ↔ Pictures

I can turn a picture into a list of numbers, and a word into a list of numbers too. If the two lists look alike, they match! Press Wake up the model to start.

Inside the model

1Chop picture into squares
2Picture → 512 numbers
3Words → 512 numbers
4Compare the lists
5Pick the winner

Each list of numbers drawn as a barcode

Word map: the picture is pulled toward its matches

Your picture
512 numbers
Best match

How the guess changed

Every time Robo looks, a snapshot lands here. Draw stroke by stroke and watch the guess change! Tap a snapshot to look at it again.

How it works

Chop the picture

The picture is cut into 49 little squares, like a puzzle. The model studies every square and how they fit together.

Picture numbers

A "picture brain" squeezes everything it saw into a list of 512 numbers. That list is the picture's secret code.

Word numbers

A "word brain" turns each word into its own list of 512 numbers, in the very same secret code.

Compare

Multiply the two lists number by number and add it all up. Big total = the codes agree = good match.

Winner

The word whose code agrees most with the picture wins. That's how the model "reads" a picture with words.

What just happened?

For grown-ups: what's really going on

This runs OpenAI's CLIP (ViT-B/32, 8-bit quantised, via Transformers.js + ONNX Runtime WebAssembly) entirely in the browser. Model: Xenova/clip-vit-base-patch32.

  • Two encoders, one space: a Vision Transformer splits the 224×224 image into 7×7 patches of 32 px and produces an image embedding; a Transformer text encoder produces a text embedding. Both are projected to 512 dimensions and L2-normalised.
  • Contrastive training: trained on ~400M (image, caption) pairs so matching pairs have high cosine similarity and mismatched pairs low (InfoNCE loss).
  • Zero-shot classification: similarity s = ⟨u, v⟩ for each label prompt, then softmax(100 · s). Raw cosines are small (≈0.2–0.3); the learned temperature (logit scale 100) turns small gaps into confident probabilities. Prompt templates such as “a drawing of a …” noticeably change results.
  • The barcodes are the actual 512-d vectors; the "agreement" strip is the element-wise product whose sum is the cosine similarity.
  • The map places words by PCA of their text embeddings; the picture is drawn at the probability-weighted average of word positions. (Projecting image and text together would mostly show CLIP's "modality gap" between images and texts.)

Further reading: CLIP (OpenAI) · CNN Explainer · Transformer Explainer.