Language Models · Word embeddings

The Word Map

A computer can't know what "cat" means. But it can learn that "cat" shows up in the same kinds of sentences as "dog". Let's turn that into a map.

Here are 42 words, scattered at random. Drag each word to where you think it belongs. Put friends close together! Then I'll try to learn my own map.

Your guess

How well the map groups friends (nearest 3 neighbours in the same group)
Robo's map scoreYour guessMistakes (loss), lower is betterYou changed something

Checkpoints — Robo's map saved along the way. Tap one to look at it.

They appear after 0, 300, 1000, 2000, 4000 and 8000 sentences.

How it works

Read sentences

Robo reads thousands of tiny sentences like "the cat chased the mouse" and "she ate some cake".

Guess the neighbours

Pick a word. Can its number-list tell which words sit next to it? Right guess: pull them closer. Wrong guess: push apart.

Squash to a map

Each word has 12 numbers. We squash them down to 2, so you can see them as dots on a flat map.

Directions mean things

The step from "man" to "woman" points the same way as "king" to "queen". That's word maths!

What just happened?

For grown-ups: what's really going on

This is a miniature word2vec trained from scratch in your browser. Nothing is downloaded.

  • Embedding: each of the ~80 vocabulary words (42 on the map plus function words) has a learned vector in ℝ¹². Words in the same group are generated from the same sentence templates, so they share contexts.
  • Skip-gram with negative sampling (SGNS): for every word and each neighbour within ±2 positions, the model raises σ(w·c) for the true context word and lowers it for 5 randomly drawn words. Plain SGD, learning rate decaying from 0.06. About 8000 sentences (≈ 280k word pairs) take a fraction of a second.
  • Cosine similarity measures closeness (the angle between vectors, not their length). The "map score" is the share of each word's 3 cosine-nearest neighbours that belong to its own group. Random vectors score about 20%.
  • The 2D picture uses PCA (power iteration on the covariance of the unit-length vectors). It is only a shadow of the 12D space, so some distances get distorted.
  • Analogies: king − man + woman is computed on normalised vectors and the nearest word (excluding the inputs) is returned. It works here because the corpus pairs king/man with "he" and queen/woman with "she". Real analogy results are much noisier than the famous examples.
  • Bias: embeddings absorb whatever patterns the text has. Train on real news or web text and the same maths yields stereotypes (for example, occupations drifting toward one gender). Here Robo only learned "he" and "she" because we wrote those sentences.
  • Jumbling the words destroys co-occurrence structure, so the score returns to chance. Modern LLMs learn context-dependent vectors (Transformers), but the first layer is still an embedding table like this one.