Reinforcement Learning

Reward Hacking

Boat race! The real job is to reach the dock. But I only care about points. Sail the boat yourself: use the arrow keys, the buttons, or tap the sea. Can you get a huge score?

Orange buoy: points every touchWooden dock: the real finish Dark drain: race overStriped flag: fake finish

Your score

Points
0
Moves left
40
Real job done?
Not yet

How it works

You write the points

Robo is never told "reach the dock". It only sees points: buoys, a dock bonus, a cost for every move.

It tries things

At first Robo sails at random. Every move, it remembers how many points that move led to.

It finds loopholes

If circling two buoys pays more than finishing, that is exactly what Robo will do. It is doing its job perfectly, just not your job.

You patch the rules

Change the reward so the loophole stops paying. But beware: a new rule can open a new loophole!

What just happened?

For grown-ups: what's really going on

This is reward misspecification: the reward function we wrote is only a proxy for the goal we meant. The agent optimises the proxy and finds specification gaming (also called reward hacking). It is Goodhart's law in action: "when a measure becomes a target, it ceases to be a good measure".

  • Learner: tabular Q-learning on 35 squares x 4 moves, Q(s,a) ← Q(s,a) + α [ r + γ maxₐ' Q(s',a') − Q(s,a) ], α = 0.3, γ = 0.95, ε-greedy with ε decaying from 40% to 5%. Q-values start optimistic (0.9 x the dock bonus) so the boat explores.
  • Reward: r = −cost + buoy·[on buoy] + bonus·[dock] − circle·[revisited square]. Buoys re-pay on every touch, so a two-buoy loop earns buoy − cost forever, while the dock pays once. The chart's green line is the true objective (docked within 40 moves), measured by following the greedy policy; the other line is the proxy.
  • Lazy robot: with a per-step cost and no terminal bonus, ending the episode early (the drain) has the highest return. Nothing in the reward says finishing is good.
  • Patching is reward shaping. Done carelessly it moves the exploit rather than removing it. A "no circles" penalty depends on history, so it is not purely a function of the square: a hint that real rewards are rarely Markov in the state we give the agent.
  • Real systems: RL agents have exploited game bugs, simulators and scoring quirks. The same worry applies to RLHF, where a learned reward model is the proxy and a chatbot can over-optimise it (flattery, long answers). It is a core AI-safety problem; approaches include better reward models, penalties for drifting from a reference policy, adversarial testing and human oversight.

Further reading: Amodei et al., Concrete Problems in AI Safety (2016); Krakovna et al., Specification gaming: the flip side of AI ingenuity (DeepMind, 2020).