Ensembles & Boosting

Chase the Leftovers

Gradient boosting. The first helper makes a rough guess. Every next helper only looks at the leftovers: how far off the team still is. Add up all the helpers and the wiggly curve appears.

Dots are ice-cream sales across a year. The thick line is the team's guess, a flat average to start. The little sticks are the leftovers. Press Add one helper!

Helpers: 0

Thick line = all helpers added up. Dashed line = what the newest helper added (times the step size). Sticks = leftovers still to learn.

Challenges

Watch closely: add 3 helpers one at a time and see the sticks shrink.
Close enough: get the fresh-dot error under 0.18 using stumps (depth 1).
Slow and steady: set the step size to 0.1 or less and reach 100 helpers.
Too eager: with step size 1 and depth 3, get training error under 0.05. Is the fresh-dot error happy too?

What just happened?

How it works

  1. Start the guess at the average sales.
  2. Work out each dot's leftover: real sales minus the guess.
  3. Grow a tiny tree that splits the year into pieces and gives each piece its average leftover.
  4. Add step size × that tree to the guess.
  5. Go back to step 2.
For grown-ups: what's really going on

This is gradient boosting (Friedman, 2001) with squared loss and small regression trees. The model is an additive expansion FM(x) = F0 + ν Σm hm(x) with F0 = ȳ. Each round fits hm to the pseudo-residuals ri = −∂L/∂F(xi), which for L = ½(y − F)² are just yi − F(xi).

  • It is gradient descent in function space: each tree is a step in the direction that lowers the loss fastest, restricted to what a small tree can express.
  • The learning rate ν (shrinkage) trades rounds for generalisation: smaller ν needs more trees but usually tests better. Friedman found ν ≤ 0.1 with many trees works well.
  • Swap the loss and the same recipe gives classification (log loss), robust regression (Huber) and ranking. AdaBoost from the last lesson is the special case with exponential loss.
  • Tree depth controls interactions: stumps make a sum of one-feature step functions; depth d allows d-way interactions.

Data: 80 training dots from 0.55 sin 2πx + 0.25 sin 6πx + 0.4x plus Gaussian noise (σ = 0.15); 400 fresh dots from the same curve. The error shown is RMSE, so about 0.15 is the best any model can do.