Mouse Finds Cheese
I'm a hungry little mouse, but I don't know the maze! Cheese is yummy, traps go ouch. Press Play and watch me learn by trying.
How's it going?
How it works
Try stuff
At first the mouse knows nothing, so it wanders around randomly. Bonk! Wall. Bonk!
Treats & ouches
Cheese gives a big treat (+10). A trap gives a big ouch (−10). Every step costs a tiny bit, so slow is bad too.
Remember
Each square keeps a score for each arrow: "going this way usually ends well". Green squares are good places to be.
Follow the best arrows
The good news spreads backwards from the cheese, square by square, until there's a path of arrows from home all the way to cheese.
What just happened?
- The mouse started knowing nothing and wandered randomly.
- Cheese gave a treat, traps gave an ouch — nobody told it where to go.
- Each square remembered a score for each arrow, and good news spread backwards from the cheese.
- The checkpoints show the same mouse brain after 1, 10, 50… tries: more practice, more green, better arrows.
- Change the maze and the old memories go wrong — it has to learn again.
For grown-ups: what's really going on
This is tabular Q-learning (Watkins, 1989) on a small grid-world Markov Decision Process. Nothing is faked: every step you see is one real update.
- State = which square the mouse is on (64 states). Actions = up / right / down / left. Walls and edges make the mouse stay put.
- Rewards: cheese
+10(episode ends), trap−10(episode ends), each step−0.1, bumping into a wall−0.3. Episodes also end after 150 steps. - Update rule:
Q(s,a) ← Q(s,a) + α · [ r + γ · maxₐ' Q(s',a') − Q(s,a) ]. "Learning speed" is α, "Patience" is the discount factor γ. - Policy: ε-greedy. With probability ε ("Curiosity") take a random action, otherwise the action with the highest Q (ties broken randomly). This is the classic exploration vs exploitation trade-off.
- The square colour is
V(s) = maxₐ Q(s,a); arrow length is each action's Q relative to the others in that square. The dashed line is the greedy policy rolled out from the start. - Try the Cliff preset with high curiosity: Q-learning is off-policy, so it learns the risky shortest path along the edge even though exploring there keeps causing accidents. (SARSA, an on-policy cousin, would learn a safer detour.)
Further reading: Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed.), chapter 6 — free online.