Language Models

Word Chopper

Computers can't read letters. They chop text into little pieces called tokens and turn every piece into a number. Type anything and watch it get chopped!

Type something in the box! I'll chop it into pieces and give every piece its own number.

Chop it!

0 pieces

Watch Robo invent chunks

Step 0

Robo starts with only letters. Then, over and over, it finds the two pieces that sit together most often in its stories and glues them into one new piece.

Watch these words get glued

Your sentence at each step (tap one)

Robo's chunk book (newest first)

Words as points

After chopping, every word gets a list of numbers that describes which words it hangs out with. Words with similar friends end up close together on this map.

What's happening inside

Point at a word on the map (or tap it).

Word math

How it works

Chop

Text gets cut into pieces. Not too big (Robo couldn't learn every word ever), not too small (too many pieces).

Glue the buddies

Robo starts with letters and keeps gluing the two pieces that are seen together the most: t + h → th, th + e → the.

Number them

Every piece in Robo's chunk book has a number. Computers only work with numbers, so "the" might be piece #262.

Place them

Each word also gets a spot on a map, near words used in the same way: cat near dog, red near blue.

What just happened?

For grown-ups: what's really going on
  • Byte-Pair Encoding (BPE) (Sennrich et al., 2016; used by GPT-2/3/4). Pre-tokenise with a GPT-2-style regex (a leading space stays attached to the word, shown as “·”), start from a base vocabulary of 96 characters, then repeatedly count adjacent symbol pairs (weighted by word frequency) and merge the most frequent pair into a new symbol. Robo learns up to 500 merges from the ~3,500-word built-in corpus, live in your browser. Encoding applies the learned merges in rank order.
  • Why sub-words? A character vocabulary makes sequences long. A word vocabulary is huge and has out-of-vocabulary words (shown as [?]). BPE trades off between the two and never hits an unknown token, since it can always fall back to characters (real GPT-2 goes down to bytes).
  • Real GPT-2 mode loads the actual Xenova/gpt2 tokenizer (50,257 tokens, byte-level BPE) through Transformers.js. Only the tokenizer files download (about 2 MB), not the model.
  • Word map. For each word we count neighbours within ±2 words inside a sentence, convert counts to PPMI (positive pointwise mutual information, with context smoothing α = 0.75), and L2-normalise. That is a classic count-based distributional embedding (“you shall know a word by the company it keeps”). The 2D picture is a live t-SNE (perplexity 10, exact gradients) of those vectors. Distances in 2D are only roughly faithful, so neighbours and word math are computed in the full space with cosine similarity.
  • Word math is the vector-offset analogy test from word2vec (Mikolov et al., 2013): argmax cos(v, king − man + woman), excluding the three input words. It works here because the small “Word Garden” corpus uses gender words (his/her, he/she) and royal words (crown, castle) consistently. On real corpora the effect is noisier.
  • Real LLMs learn embeddings by gradient descent as part of the network instead of counting, but the intuition carries over. See the Transformer Explainer (Polo Club) for how token embeddings flow into attention.