Ensembles & Boosting

Learn from Mistakes

Each tiny helper makes just one straight cut. After each round, the dots it got wrong grow bigger so the next helper pays more attention.

Purple dots are inside a circle, orange are outside. One straight cut cannot do it. Press Add one helper and watch the mistakes grow!

Round 0

Bigger dot = the learner cares more about it. Thick ring = the newest helper got it wrong. Dashed line = the newest helper's cut. Background = what all helpers say together.

Challenges

Take it step by step: add 3 helpers one at a time.
Spot the hard one: get a dot to be at least twice as big as at the start.
Strong together: reach 90% on fresh dots, even though each helper alone makes only one cut.
Keep going: reach 40 rounds.

What just happened?

How it works

  1. Give every dot the same weight.
  2. Find the single cut (a stump) with the lowest weighted mistake rate.
  3. Give that helper a vote size from its error: fewer mistakes means a louder vote.
  4. Multiply the weights of the dots it got wrong up, and the right ones down, then rescale.
  5. Repeat. The final answer is the sign of the sum of all votes.
For grown-ups: what's really going on

This is discrete AdaBoost (Freund and Schapire) with depth-1 decision stumps as weak learners. For round t with weighted error εt: αt = ½ ln((1 − εt) / εt), then wi ← wi · exp(−αt yi ht(xi)), normalised to sum 1. The final model is H(x) = sign(Σ αt ht(x)).

  • Boosting reduces bias: it builds a complex boundary out of simple pieces. Each stump looks at one axis, so the final model is a sum of one-feature functions. A circle (x² + y²) is exactly such a sum, which is why this puzzle suits stumps.
  • It is a stagewise additive model minimising exponential loss Σ exp(−y F(x)), which is why misclassified points get exponentially growing weight.
  • Training error can reach zero while test error keeps improving on clean data, a famous observation about margins. With noisy labels, though, the growing weights chase the wrong labels; the next lesson shows this.
  • Errors are clamped strictly between 0 and ½ so α stays finite.

Data: 90 random dots in the unit square, class 1 inside a circle of radius 0.33 around the centre, no label noise. Tested on 500 fresh dots.