Hot or Cold Words
A language model never knows the next word for sure. It gives every word a chance, then rolls a dice. The temperature dial decides how wild that dice is.
I read some little stories and counted which words come after which. Look at the bars: those are my chances for the next word. Press Pick next word and I will roll my word dice!
How to play: pick some starting words, press Pick next word a few times, then turn the temperature dial and see the bars change shape.
The story so far
Chances for the next word
Same start, 5 tries
Write the same start five times with your settings. Are the stories all the same? Do they make sense? Words that never follow each other in the stories get a wavy line.
Challenges
What just happened?
- The model looked at the last two words and listed every word it knows with a chance for each. The chances add up to 100%.
- It picks the next word like a lucky dip: big bars are picked often, small bars sometimes.
- Cold makes the biggest bar even bigger. At 0 it always picks the top word, so every try is the same, and it can get stuck in loops.
- Hot squashes the bars flatter, so odd words sneak in. Fun at first, then nonsense.
- Top-k and top-p throw the tiny bars out of the hat before the dice is rolled. That lets you be a bit hot and still make sense.
- Big chat models do exactly this, one word piece at a time, with many thousands of possible pieces.
How it works
- We read five little stories (about 4,000 words) and count which word comes after each pair of words, and after each single word.
- For the next word, we mix those counts: mostly "after these two words", some "after this one word", and a tiny bit of "how common is this word anyway".
- The temperature re-shapes the chances: each chance is raised to the power 1 ÷ temperature, then they are scaled to add to 100% again.
- Top-k keeps only the k biggest bars. Top-p keeps the biggest bars until they add up to p. The kept bars are scaled up to 100%.
- We roll a dice on what is left, add the word to the story, and repeat.
For grown-ups: what's really going on
The model is an interpolated trigram language model: P(w | u,v) = 0.6·P₃(w|u,v) + 0.37·P₂(w|v) + 0.03·P₁(w) with maximum-likelihood counts (weights renormalised when the trigram context is unseen). The unigram term gives every vocabulary word a small non-zero probability, which is what high temperature brings to the surface.
- Temperature: treating
z = log Pas logits, sampling usessoftmax(z / T), i.e.q ∝ P1/T. AsT → 0this tends to greedy decoding (argmax);T = 1is the model's own distribution;T → ∞tends to uniform. HereT ≤ 0.01is treated as exact argmax. - Top-k sampling (Fan et al., 2018) truncates to the k most likely tokens and renormalises.
- Nucleus (top-p) sampling (Holtzman et al., 2019) keeps the smallest set of most likely tokens whose cumulative probability is at least p, so the candidate set adapts: narrow when the model is confident, wide when it is not. Filters are applied after temperature, as in most LLM libraries.
- "Different-ness" is distinct-2 (unique bigrams ÷ all bigrams across the five runs). "Makes sense" is a crude fluency proxy: the share of adjacent word pairs that occur in the training text. Real evaluations use perplexity under a stronger model or human judgement.
Greedy decoding maximises per-step likelihood but is prone to repetition loops; this diversity–coherence trade-off is why production LLMs typically sample with T ≈ 0.7–1 plus top-p ≈ 0.9.