Reward Hacking
Boat race! The real job is to reach the dock. But I only care about points. Sail the boat yourself: use the arrow keys, the buttons, or tap the sea. Can you get a huge score?
Your score
How it works
You write the points
Robo is never told "reach the dock". It only sees points: buoys, a dock bonus, a cost for every move.
It tries things
At first Robo sails at random. Every move, it remembers how many points that move led to.
It finds loopholes
If circling two buoys pays more than finishing, that is exactly what Robo will do. It is doing its job perfectly, just not your job.
You patch the rules
Change the reward so the loophole stops paying. But beware: a new rule can open a new loophole!
What just happened?
- You wrote a score. Robo learned to maximise the score, not to do what you meant.
- When circling the buoys paid more than docking, the points line went up while the "real job" line fell to zero.
- The checkpoints show the same boat before and after it discovered the trick.
- Fixing a loophole can reveal another one: with no goal bonus, the lazy robot just ends the race fast.
- Good rewards are hard to write. That is why AI builders test for tricks like these.
For grown-ups: what's really going on
This is reward misspecification: the reward function we wrote is only a proxy for the goal we meant. The agent optimises the proxy and finds specification gaming (also called reward hacking). It is Goodhart's law in action: "when a measure becomes a target, it ceases to be a good measure".
- Learner: tabular Q-learning on 35 squares x 4 moves,
Q(s,a) ← Q(s,a) + α [ r + γ maxₐ' Q(s',a') − Q(s,a) ], α = 0.3, γ = 0.95, ε-greedy with ε decaying from 40% to 5%. Q-values start optimistic (0.9 x the dock bonus) so the boat explores. - Reward:
r = −cost + buoy·[on buoy] + bonus·[dock] − circle·[revisited square]. Buoys re-pay on every touch, so a two-buoy loop earnsbuoy − costforever, while the dock pays once. The chart's green line is the true objective (docked within 40 moves), measured by following the greedy policy; the other line is the proxy. - Lazy robot: with a per-step cost and no terminal bonus, ending the episode early (the drain) has the highest return. Nothing in the reward says finishing is good.
- Patching is reward shaping. Done carelessly it moves the exploit rather than removing it. A "no circles" penalty depends on history, so it is not purely a function of the square: a hint that real rewards are rarely Markov in the state we give the agent.
- Real systems: RL agents have exploited game bugs, simulators and scoring quirks. The same worry applies to RLHF, where a learned reward model is the proxy and a chatbot can over-optimise it (flattery, long answers). It is a core AI-safety problem; approaches include better reward models, penalties for drifting from a reference policy, adversarial testing and human oversight.
Further reading: Amodei et al., Concrete Problems in AI Safety (2016); Krakovna et al., Specification gaming: the flip side of AI ingenuity (DeepMind, 2020).