Puppy Trainer
This puppy knows lots of tricks, good and naughty. Tap the puppy that deserves a treat!
WHAT THE PREDICTOR THINKS EARNS A TREAT
HOW OFTEN THE PUPPY PICKS EACH TRICK
Checkpoints — tap to watch the puppy back then
Saved during practice.
How it works
You judge
Two tricks. You pick the better one.
Predictor learns
It learns to guess which tricks you'd treat.
Puppy practises
Lots of practice, with pretend treats from the predictor.
What just happened?
- Your few treats taught the predictor, and the predictor then trained the puppy thousands of times.
- Chatbots learn manners the same way: people pick better answers, then the bot practises.
For grown-ups: what's really going on
This is RLHF (reinforcement learning from human feedback), the recipe used to align chat models like InstructGPT.
- Preferences: each click is a pairwise comparison (A ≻ B).
- Reward model: linear over features (8 trick one-hots + "fast"), trained with the Bradley–Terry loss
−log σ(r(A) − r(B)). Accuracy is measured on 60 held-out pairs scored by the auto-human's hidden taste. - Policy: softmax over 16 actions (trick × speed). Updated with REINFORCE and a running-mean baseline:
θ ← θ + η (r − b) ∇log π(a). - KL penalty: the slider pulls the logits back toward the original policy, approximating PPO's KL term. Without it, a flawed reward model gets exploited: reward hacking (try the buggy predictor).
Further reading: Ouyang et al., “Training language models to follow instructions with human feedback” (2022).