Reinforcement Learning

Balance the Pole

Can you balance a broom on your hand? Let's try with a little cart! Press ← and → (or the push buttons) to keep the pole standing up. Then let me learn it myself!

Goal: balance for 10 seconds
How long it balanced, each trydashed line = 10 s goal · dark line = average
Checkpoints: tap one to replay that brainsame robot, different amounts of practice
Checkpoints appear here as Robo practises.

How it works

Look

Robo checks four things: which way the pole leans, how fast it's falling, and how fast the cart is rolling.

Check the cheat-sheet

For each kind of wobble, Robo's table has a score for "push left" and "push right". Bigger score = better idea.

Treat for every moment

Robo gets a tiny treat for every moment the pole stays up. When it falls, the treats stop.

Fix the scores

After each push, Robo updates the score: "that led to more treats" goes up, "that led to a fall" goes down.

What just happened?

For grown-ups: what's really going on

This is the classic CartPole control task (Barto, Sutton & Anderson, 1983; the same dynamics as Gymnasium's CartPole-v1), solved with tabular Q-learning.

  • Physics: cart mass 1 kg, pole mass 0.1 kg, pole half-length 0.5 m, push force ±10 N, Euler integration at 50 Hz (τ = 0.02 s). An episode fails if the pole tilts past ±12° or the cart leaves ±2.4 m; it succeeds at 500 steps (10 s). In "You try" mode you can also choose not to push.
  • State: the continuous state (x, ẋ, θ, θ̇) is bucketed into 1 × 4 × 8 × 10 = 320 discrete cells (cart position is ignored; the velocity, angle and angular velocity buckets span ±2 m/s, ±12°, ±2.5 rad/s).
  • Reward: +1 per step alive, 0 on failure. Update: Q(s,a) ← Q(s,a) + α [r + γ maxₐ' Q(s',a') − Q(s,a)], with γ = 0.99.
  • Schedules: with auto curiosity, ε = α = max(min, 1 − log₁₀((episode+1)/25)), where the floor is 0.01 for ε and 0.1 for α. You can switch auto off and set ε yourself.
  • When tuned offline over 8 random seeds, this setup reaches a 20-episode average of ≥ 475 steps after 212–305 episodes (roughly 100k simulation steps). Tabular methods can still forget after solving; deep RL (DQN, policy gradients) replaces the table with a neural network that generalises between buckets.
  • The heatmap is a slice of the Q-table: angle × angular velocity at the cart's current velocity bucket. The colour shows argmax, and the strength shows |Q(right) − Q(left)|.

Further reading: Sutton & Barto, Reinforcement Learning: An Introduction, ch. 6 & 9; Gymnasium CartPole docs.