Reinforcement Learning

Mouse Finds Cheese

I'm a hungry little mouse, but I don't know the maze! Cheese is yummy, traps go ouch. Press Play and watch me learn by trying.

Checkpoints: tap to see the brain back thentap again to go back to live
Snapshots appear as the mouse practises.
Good spot (close to cheese) Bad spot (near traps) Best arrow = mouse's favorite move Footsteps this try

How's it going?

Tries
0
Steps now
0
Cheese
0
Traps hit
0
Steps needed each try (lower = smarter)

How it works

Try stuff

At first the mouse knows nothing, so it wanders around randomly. Bonk! Wall. Bonk!

Treats & ouches

Cheese gives a big treat (+10). A trap gives a big ouch (−10). Every step costs a tiny bit, so slow is bad too.

Remember

Each square keeps a score for each arrow: "going this way usually ends well". Green squares are good places to be.

Follow the best arrows

The good news spreads backwards from the cheese, square by square, until there's a path of arrows from home all the way to cheese.

What just happened?

For grown-ups: what's really going on

This is tabular Q-learning (Watkins, 1989) on a small grid-world Markov Decision Process. Nothing is faked: every step you see is one real update.

  • State = which square the mouse is on (64 states). Actions = up / right / down / left. Walls and edges make the mouse stay put.
  • Rewards: cheese +10 (episode ends), trap −10 (episode ends), each step −0.1, bumping into a wall −0.3. Episodes also end after 150 steps.
  • Update rule: Q(s,a) ← Q(s,a) + α · [ r + γ · maxₐ' Q(s',a') − Q(s,a) ]. "Learning speed" is α, "Patience" is the discount factor γ.
  • Policy: ε-greedy. With probability ε ("Curiosity") take a random action, otherwise the action with the highest Q (ties broken randomly). This is the classic exploration vs exploitation trade-off.
  • The square colour is V(s) = maxₐ Q(s,a); arrow length is each action's Q relative to the others in that square. The dashed line is the greedy policy rolled out from the start.
  • Try the Cliff preset with high curiosity: Q-learning is off-policy, so it learns the risky shortest path along the edge even though exploring there keeps causing accidents. (SARSA, an on-policy cousin, would learn a safer detour.)

Further reading: Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed.), chapter 6 — free online.