Reinforcement Learning

Lucky Slot Machines

Five slot machines. One of them secretly pays the most. You get a few pulls: which machine do you stick with, and when do you try a new one?

Welcome to the casino! Every machine pays a different amount, and nobody tells you which is best. You have 30 pulls. Tap a machine to pull it!

Your turn

Pulls left
30
Your coins
0
Best possible
?
Each machine shows how many times you pulled it and what it paid on average so far.

Watch Robo learn

Robo's notebook

Bar = average coins Robo has seen from each machine. Star = Robo's favourite.
Pulls
0
Coins
0
Per pull
0.0
Best possible
0.0
RoboBest possibleCasino changecoins per pull, last 30

Strategy race

Three robots play the very same machines with the very same luck.
Never exploreCurious 10%Always random
Never
0
Curious
0
Random
0

How it works

Pull one

Robo picks a machine and pulls. Sometimes it pays, sometimes nothing.

Write it down

Each machine has a running average: how much it usually pays.

Cash in

Most of the time Robo picks the machine with the best average so far.

Stay curious

Sometimes Robo rolls a dice and tries a random machine, in case a better one is hiding.

What just happened?

For grown-ups: what's really going on

This is the multi-armed bandit problem: the simplest reinforcement-learning setting, with one state and several actions (arms). It isolates the exploration vs exploitation trade-off.

  • Machines: each arm pays a prize with a fixed win probability, with the prize size varying by up to the noise setting (unbiased). Mean payout is p x prize, so the true best is known to the page but hidden from the learner.
  • Estimate: sample average Q(a) += (r - Q(a)) / N(a), with the step floored at 0.1 so old data fades. That floor is a constant step-size update, which is what lets Robo track a changing casino (a non-stationary bandit).
  • Policy: epsilon-greedy. With probability eps pick a random arm, otherwise argmax Q (ties random). Q starts at 0, so pure greedy locks onto the first machine that pays anything. The decaying option uses eps / (1 + t/40), floored at 3% of eps.
  • Regret is the gap between the best possible total and what you earned, sum(mu* - r_t). The "best possible" line on the chart is mu*. Greedy has linear regret when it locks onto a wrong arm, and fixed-eps has linear regret from endless exploring; decaying eps can do better.
  • The race uses common random numbers: every step draws one payout per machine, and each strategy sees the payout for the arm it chose.
  • Smarter methods exist: UCB adds an optimism bonus to rarely tried arms, and Thompson sampling samples from a belief about each arm and plays the best sample. Both explore in a more targeted way than a dice roll.

Further reading: Sutton & Barto, Reinforcement Learning: An Introduction, chapter 2 (free online).