Balance the Pole
Can you balance a broom on your hand? Let's try with a little cart! Press ← and → (or the push buttons) to keep the pole standing up. Then let me learn it myself!
How it works
Look
Robo checks four things: which way the pole leans, how fast it's falling, and how fast the cart is rolling.
Check the cheat-sheet
For each kind of wobble, Robo's table has a score for "push left" and "push right". Bigger score = better idea.
Treat for every moment
Robo gets a tiny treat for every moment the pole stays up. When it falls, the treats stop.
Fix the scores
After each push, Robo updates the score: "that led to more treats" goes up, "that led to a fall" goes down.
What just happened?
- Balancing is hard — you probably felt it! Tiny pushes at the right moment matter.
- Robo started with an empty cheat-sheet and fell over almost instantly.
- Nobody told Robo the rules. It only got a treat for every moment the pole stayed up.
- After a few hundred falls, the scores in its table pointed the right way and it could balance for the full 10 seconds.
- The checkpoints show the same robot at different moments of practice. Practice is what made it good.
For grown-ups: what's really going on
This is the classic CartPole control task (Barto, Sutton & Anderson, 1983; the same dynamics as Gymnasium's CartPole-v1), solved with tabular Q-learning.
- Physics: cart mass 1 kg, pole mass 0.1 kg, pole half-length 0.5 m, push force ±10 N, Euler integration at 50 Hz (
τ = 0.02 s). An episode fails if the pole tilts past ±12° or the cart leaves ±2.4 m; it succeeds at 500 steps (10 s). In "You try" mode you can also choose not to push. - State: the continuous state
(x, ẋ, θ, θ̇)is bucketed into1 × 4 × 8 × 10 = 320discrete cells (cart position is ignored; the velocity, angle and angular velocity buckets span ±2 m/s, ±12°, ±2.5 rad/s). - Reward: +1 per step alive, 0 on failure. Update:
Q(s,a) ← Q(s,a) + α [r + γ maxₐ' Q(s',a') − Q(s,a)], withγ = 0.99. - Schedules: with auto curiosity,
ε = α = max(min, 1 − log₁₀((episode+1)/25)), where the floor is 0.01 for ε and 0.1 for α. You can switch auto off and set ε yourself. - When tuned offline over 8 random seeds, this setup reaches a 20-episode average of ≥ 475 steps after 212–305 episodes (roughly 100k simulation steps). Tabular methods can still forget after solving; deep RL (DQN, policy gradients) replaces the table with a neural network that generalises between buckets.
- The heatmap is a slice of the Q-table: angle × angular velocity at the cart's current velocity bucket. The colour shows
argmax, and the strength shows|Q(right) − Q(left)|.
Further reading: Sutton & Barto, Reinforcement Learning: An Introduction, ch. 6 & 9; Gymnasium CartPole docs.