Learning from Examples

Twenty Questions Trees

A decision tree plays twenty questions: "Is it left of this line? Is it above that one?" Grow one question at a time, then plant a whole forest of trees that vote together.

See the red, blue and yellow shapes? I want to sort them with as few yes/no questions as I can. Press Ask one question and watch me draw my first line!

Tap to add shapes · point at any spot to see how the trees decide · hollow shapes are test shapes the trees never learn from

The question tree

Each box asks a question. Its bar shows how mixed the shapes inside are: a long bar = very mixed, empty bar = all the same.

The forest

Switch to Forest and press Plant a forest. Point at the big map to see how each little tree votes.

Snapshots while learning

A photo of the map after 1, 2, 4, 8… questions. Tap one to look back.

How it works

Try every question

The tree tries every possible straight line: "left or right of here?", "above or below here?"

Pick the tidiest split

It keeps the question that leaves each side the least mixed up.

Ask again inside

Each side gets its own new question, until every box is tidy or we run out of questions.

Forest votes

Many slightly different trees each guess. The most popular answer wins — fewer silly mistakes!

What just happened?

For grown-ups: what's really going on

This is a CART decision tree and a random forest on two features.

  • Each split is an axis-aligned threshold chosen to minimise weighted Gini impurity G = 1 − Σ pₖ² of the two children ("how mixed is this box"). The tree here grows best-first: each click splits the leaf with the biggest impurity reduction.
  • Max depth limits complexity. Deep trees have low bias but high variance: practice (training) accuracy → 100% while test accuracy drops — overfitting.
  • A random forest trains each tree on a bootstrap sample (draw n points with replacement) and considers a random feature subset at each split (here: one of the two axes). Predictions average the trees' class probabilities (bagging), which cuts variance.
  • The small multiples show each tree's own decision map; the big map is the averaged vote.

Further reading: Breiman (2001) "Random Forests" · R2D3 "A visual introduction to machine learning" · ISLR ch. 8.