Lucky Slot Machines
Five slot machines. One of them secretly pays the most. You get a few pulls: which machine do you stick with, and when do you try a new one?
Welcome to the casino! Every machine pays a different amount, and nobody tells you which is best. You have 30 pulls. Tap a machine to pull it!
Your turn
Watch Robo learn
Robo's notebook
Strategy race
How it works
Pull one
Robo picks a machine and pulls. Sometimes it pays, sometimes nothing.
Write it down
Each machine has a running average: how much it usually pays.
Cash in
Most of the time Robo picks the machine with the best average so far.
Stay curious
Sometimes Robo rolls a dice and tries a random machine, in case a better one is hiding.
What just happened?
- Nobody told you or Robo which machine was best. You both had to find out by playing.
- If you only pull your favourite, you might stick with a machine that was just lucky early on.
- If you only try random machines, you never cash in on what you learned.
- A little curiosity (about 10%) beat both extremes in the race.
- When the casino changed the machines, only the curious robots noticed.
For grown-ups: what's really going on
This is the multi-armed bandit problem: the simplest reinforcement-learning setting, with one state and several actions (arms). It isolates the exploration vs exploitation trade-off.
- Machines: each arm pays a prize with a fixed win probability, with the prize size varying by up to the noise setting (unbiased). Mean payout is
p x prize, so the true best is known to the page but hidden from the learner. - Estimate: sample average
Q(a) += (r - Q(a)) / N(a), with the step floored at 0.1 so old data fades. That floor is a constant step-size update, which is what lets Robo track a changing casino (a non-stationary bandit). - Policy: epsilon-greedy. With probability eps pick a random arm, otherwise argmax Q (ties random). Q starts at 0, so pure greedy locks onto the first machine that pays anything. The decaying option uses
eps / (1 + t/40), floored at 3% of eps. - Regret is the gap between the best possible total and what you earned,
sum(mu* - r_t). The "best possible" line on the chart ismu*. Greedy has linear regret when it locks onto a wrong arm, and fixed-eps has linear regret from endless exploring; decaying eps can do better. - The race uses common random numbers: every step draws one payout per machine, and each strategy sees the payout for the arm it chose.
- Smarter methods exist: UCB adds an optimism bonus to rarely tried arms, and Thompson sampling samples from a belief about each arm and plays the best sample. Both explore in a more targeted way than a dice roll.
Further reading: Sutton & Barto, Reinforcement Learning: An Introduction, chapter 2 (free online).