The Careful Booster
This is the idea behind XGBoost, a famous booster that wins lots of competitions. Two boosters race on the same noisy dots. The wild one grows big trees and believes every dot. The careful one has brakes: it shrinks its guesses, skips splits that do not help much, and ignores tiny groups.
Both boosters use the same trees and the same step size. Only the brakes are different. Press Add 10 trees and watch the two curves.
Trees: 0
Challenges
What just happened?
- Both boosters chase the leftovers, like in the last lesson. Big trees can chase them all the way into the noise.
- The careful booster pulls every leaf's guess a little towards zero (λ), so a leaf built from only a few dots cannot shout.
- It only keeps a split if it lowers the error by at least γ. Splits that just chase one noisy dot get cut off.
- It never makes a group smaller than the minimum. Result: simpler trees that do better on dots they have never seen.
How it works
- For every dot, work out how wrong the team is (the gradient) and how sure that number is (the hessian).
- Try every split. Score it with a formula that already includes the λ shrink.
- If the best split's score is less than γ, stop and make a leaf instead.
- Each leaf's value is the summed gradient divided by (summed hessian + λ).
- Add step size × the tree to the team. Repeat.
For grown-ups: what's really going on
XGBoost (Chen and Guestrin, 2016) is gradient boosting with a regularised objective Σ l(yi, ŷi) + Σk [γTk + ½λ‖wk‖²], where T is the number of leaves and w the leaf weights. Each tree is fitted with a second-order Taylor expansion of the loss using gradients gi and hessians hi.
- Optimal leaf weight:
w* = −G / (H + λ). Split gain:½[GL²/(HL+λ) + GR²/(HR+λ) − G²/(H+λ)] − γ. A split is kept only if the gain is positive, which is pre-pruning by γ. min_child_weightis the minimum hessian sum in a child. For squared loss h = 1, so it is just a minimum number of dots.- This demo uses exact greedy split finding on one feature. Real XGBoost adds column and row subsampling, histogram/approximate splits, sparsity-aware default directions, and a fast parallel C++ core. LightGBM is a close cousin with leaf-wise growth and histogram binning.
- The wild booster is the same code with λ = γ = 0 and min child weight 1, so any difference you see comes from the regulariser alone.
Data: 80 training dots from a wavy curve plus Gaussian noise (σ = 0.15), 400 fresh dots. Error is RMSE; about 0.15 is the noise floor.