RLHF · learning from people

Puppy Trainer

This puppy knows lots of tricks, good and naughty. Tap the puppy that deserves a treat!

WHAT THE PREDICTOR THINKS EARNS A TREAT

HOW OFTEN THE PUPPY PICKS EACH TRICK

Checkpoints — tap to watch the puppy back then

Saved during practice.

How it works

You judge

Two tricks. You pick the better one.

Predictor learns

It learns to guess which tricks you'd treat.

Puppy practises

Lots of practice, with pretend treats from the predictor.

What just happened?

For grown-ups: what's really going on

This is RLHF (reinforcement learning from human feedback), the recipe used to align chat models like InstructGPT.

  • Preferences: each click is a pairwise comparison (A ≻ B).
  • Reward model: linear over features (8 trick one-hots + "fast"), trained with the Bradley–Terry loss −log σ(r(A) − r(B)). Accuracy is measured on 60 held-out pairs scored by the auto-human's hidden taste.
  • Policy: softmax over 16 actions (trick × speed). Updated with REINFORCE and a running-mean baseline: θ ← θ + η (r − b) ∇log π(a).
  • KL penalty: the slider pulls the logits back toward the original policy, approximating PPO's KL term. Without it, a flawed reward model gets exploited: reward hacking (try the buggy predictor).

Further reading: Ouyang et al., “Training language models to follow instructions with human feedback” (2022).