The Word Map
A computer can't know what "cat" means. But it can learn that "cat" shows up in the same kinds of sentences as "dog". Let's turn that into a map.
Here are 42 words, scattered at random. Drag each word to where you think it belongs. Put friends close together! Then I'll try to learn my own map.
Your guess
Checkpoints — Robo's map saved along the way. Tap one to look at it.
How it works
Read sentences
Robo reads thousands of tiny sentences like "the cat chased the mouse" and "she ate some cake".
Guess the neighbours
Pick a word. Can its number-list tell which words sit next to it? Right guess: pull them closer. Wrong guess: push apart.
Squash to a map
Each word has 12 numbers. We squash them down to 2, so you can see them as dots on a flat map.
Directions mean things
The step from "man" to "woman" points the same way as "king" to "queen". That's word maths!
What just happened?
- At first the map was random. Nobody told Robo that cats and dogs are both animals.
- Robo only saw which words sit next to each other. Words used in the same places got the same kind of numbers, so they landed close together.
- Jumble the sentences and the map falls apart: meaning comes from context.
- Chatbots start the same way. Every word becomes a list of numbers learned from lots of text.
For grown-ups: what's really going on
This is a miniature word2vec trained from scratch in your browser. Nothing is downloaded.
- Embedding: each of the ~80 vocabulary words (42 on the map plus function words) has a learned vector in ℝ¹². Words in the same group are generated from the same sentence templates, so they share contexts.
- Skip-gram with negative sampling (SGNS): for every word and each neighbour within ±2 positions, the model raises
σ(w·c)for the true context word and lowers it for 5 randomly drawn words. Plain SGD, learning rate decaying from 0.06. About 8000 sentences (≈ 280k word pairs) take a fraction of a second. - Cosine similarity measures closeness (the angle between vectors, not their length). The "map score" is the share of each word's 3 cosine-nearest neighbours that belong to its own group. Random vectors score about 20%.
- The 2D picture uses PCA (power iteration on the covariance of the unit-length vectors). It is only a shadow of the 12D space, so some distances get distorted.
- Analogies:
king − man + womanis computed on normalised vectors and the nearest word (excluding the inputs) is returned. It works here because the corpus pairs king/man with "he" and queen/woman with "she". Real analogy results are much noisier than the famous examples. - Bias: embeddings absorb whatever patterns the text has. Train on real news or web text and the same maths yields stereotypes (for example, occupations drifting toward one gender). Here Robo only learned "he" and "she" because we wrote those sentences.
- Jumbling the words destroys co-occurrence structure, so the score returns to chance. Modern LLMs learn context-dependent vectors (Transformers), but the first layer is still an embedding table like this one.