Chase the Leftovers
Gradient boosting. The first helper makes a rough guess. Every next helper only looks at the leftovers: how far off the team still is. Add up all the helpers and the wiggly curve appears.
Dots are ice-cream sales across a year. The thick line is the team's guess, a flat average to start. The little sticks are the leftovers. Press Add one helper!
Helpers: 0
Thick line = all helpers added up. Dashed line = what the newest helper added (times the step size). Sticks = leftovers still to learn.
Challenges
What just happened?
- The team starts with the plain average. Every dot has a leftover: the gap between the dot and the guess.
- A new little tree learns to predict those leftovers, not the sales themselves.
- The team adds a small slice of that tree (the step size), so the guess moves a bit closer to every dot.
- Repeat. Leftovers shrink and the curve gets wigglier. Too many big steps and it starts chasing the noise too.
How it works
- Start the guess at the average sales.
- Work out each dot's leftover: real sales minus the guess.
- Grow a tiny tree that splits the year into pieces and gives each piece its average leftover.
- Add step size × that tree to the guess.
- Go back to step 2.
For grown-ups: what's really going on
This is gradient boosting (Friedman, 2001) with squared loss and small regression trees. The model is an additive expansion FM(x) = F0 + ν Σm hm(x) with F0 = ȳ. Each round fits hm to the pseudo-residuals ri = −∂L/∂F(xi), which for L = ½(y − F)² are just yi − F(xi).
- It is gradient descent in function space: each tree is a step in the direction that lowers the loss fastest, restricted to what a small tree can express.
- The learning rate ν (shrinkage) trades rounds for generalisation: smaller ν needs more trees but usually tests better. Friedman found ν ≤ 0.1 with many trees works well.
- Swap the loss and the same recipe gives classification (log loss), robust regression (Huber) and ranking. AdaBoost from the last lesson is the special case with exponential loss.
- Tree depth controls interactions: stumps make a sum of one-feature step functions; depth d allows d-way interactions.
Data: 80 training dots from 0.55 sin 2πx + 0.25 sin 6πx + 0.4x plus Gaussian noise (σ = 0.15); 400 fresh dots from the same curve. The error shown is RMSE, so about 0.15 is the best any model can do.