Language Models & LLMs

Hot or Cold Words

A language model never knows the next word for sure. It gives every word a chance, then rolls a dice. The temperature dial decides how wild that dice is.

I read some little stories and counted which words come after which. Look at the bars: those are my chances for the next word. Press Pick next word and I will roll my word dice!

How to play: pick some starting words, press Pick next word a few times, then turn the temperature dial and see the bars change shape.

The story so far

Chances for the next word

chance now chance at normal temperature (1) thrown out of the hat the word just picked

Same start, 5 tries

Write the same start five times with your settings. Are the stories all the same? Do they make sense? Words that never follow each other in the stories get a wavy line.

    Different-ness-
    Makes sense-

    Challenges

    Ice cold: set the temperature to 0.1 or lower and press Repeat 5 times. All five stories should be exactly the same.
    Hot mess: temperature 2 or more, no top-k or top-p. Repeat 5 times: five different stories, and "makes sense" below 60%.
    Tame the heat: keep the temperature at 2 or more, but switch on top-p (60% or less) or top-k (5 or less). Repeat 5 times: at least 4 different stories that make sense 70% or more.

    What just happened?

    How it works

    1. We read five little stories (about 4,000 words) and count which word comes after each pair of words, and after each single word.
    2. For the next word, we mix those counts: mostly "after these two words", some "after this one word", and a tiny bit of "how common is this word anyway".
    3. The temperature re-shapes the chances: each chance is raised to the power 1 ÷ temperature, then they are scaled to add to 100% again.
    4. Top-k keeps only the k biggest bars. Top-p keeps the biggest bars until they add up to p. The kept bars are scaled up to 100%.
    5. We roll a dice on what is left, add the word to the story, and repeat.
    For grown-ups: what's really going on

    The model is an interpolated trigram language model: P(w | u,v) = 0.6·P₃(w|u,v) + 0.37·P₂(w|v) + 0.03·P₁(w) with maximum-likelihood counts (weights renormalised when the trigram context is unseen). The unigram term gives every vocabulary word a small non-zero probability, which is what high temperature brings to the surface.

    • Temperature: treating z = log P as logits, sampling uses softmax(z / T), i.e. q ∝ P1/T. As T → 0 this tends to greedy decoding (argmax); T = 1 is the model's own distribution; T → ∞ tends to uniform. Here T ≤ 0.01 is treated as exact argmax.
    • Top-k sampling (Fan et al., 2018) truncates to the k most likely tokens and renormalises.
    • Nucleus (top-p) sampling (Holtzman et al., 2019) keeps the smallest set of most likely tokens whose cumulative probability is at least p, so the candidate set adapts: narrow when the model is confident, wide when it is not. Filters are applied after temperature, as in most LLM libraries.
    • "Different-ness" is distinct-2 (unique bigrams ÷ all bigrams across the five runs). "Makes sense" is a crude fluency proxy: the share of adjacent word pairs that occur in the training text. Real evaluations use perplexity under a stronger model or human judgement.

    Greedy decoding maximises per-step likelihood but is prone to repetition loops; this diversity–coherence trade-off is why production LLMs typically sample with T ≈ 0.7–1 plus top-p ≈ 0.9.