Reinforcement Learning

Snake learns to eat

Steer the snake to the apples with the arrow keys. Don't bump the walls or your own tail! Then let a robot snake learn it from scratch.

Arrow keys or buttons
Apples eaten, each trybars = each try · dark line = average of last 20
Checkpoints: tap one to replay that brainsame snake, different amounts of practice
Checkpoints appear here as Robo practises.

How it works

Feel

The snake only feels a few things: is there danger ahead, left or right, and which way the apple is.

Check the cheat-sheet

For each mix of feelings, its table has scores for "straight", "left" and "right". It picks the biggest.

Treats and ouches

An apple is a big treat. Bumping a wall or its tail is a big ouch. Wandering costs a tiny bit.

Fix the scores

After every move it nudges that score toward what really happened next. Hundreds of games later, it hunts apples.

What just happened?

For grown-ups: what's really going on

Another tabular Q-learning agent, this time with a hand-designed, relative state.

  • State (11 bits → at most 2,048 cells): danger straight / right / left (relative to heading), heading (4 values), and food up / down / left / right.
  • Actions: straight, turn right, turn left. Relative actions mean the snake can never reverse into itself.
  • Reward: +10 per apple, −10 for dying, −0.01 per step. A game also ends if the snake wanders too long without eating. α = 0.3, γ = 0.9.
  • Exploration: with auto curiosity, ε decays from 0.1 to 0.005 over the first 300 games.
  • Offline in Node over 3 runs: 50-game average ≈ 3 apples after 100 games, ≈ 8 after 200, ≈ 18 after 400. The greedy policy then averages 17–19 apples. Long snakes still die by coiling into themselves, because the state can't see the body beyond one cell.
  • Deep RL versions (DQN) feed the whole grid into a neural network and can learn longer-range planning.