Ensembles & Boosting

The Careful Booster

This is the idea behind XGBoost, a famous booster that wins lots of competitions. Two boosters race on the same noisy dots. The wild one grows big trees and believes every dot. The careful one has brakes: it shrinks its guesses, skips splits that do not help much, and ignores tiny groups.

Both boosters use the same trees and the same step size. Only the brakes are different. Press Add 10 trees and watch the two curves.

Trees: 0

wild boostercareful boosterthe real pattern

Challenges

Brakes win: after 50 trees, the careful booster beats the wild one on fresh dots.
Less is more: the careful booster wins with fewer than a quarter of the wild one's leaves.
No brakes: set λ to 0, γ to 0 and the smallest group to 1. Are the two boosters now the same?
Too careful: push γ so high that the careful booster stops growing any splits at all.

What just happened?

How it works

  1. For every dot, work out how wrong the team is (the gradient) and how sure that number is (the hessian).
  2. Try every split. Score it with a formula that already includes the λ shrink.
  3. If the best split's score is less than γ, stop and make a leaf instead.
  4. Each leaf's value is the summed gradient divided by (summed hessian + λ).
  5. Add step size × the tree to the team. Repeat.
For grown-ups: what's really going on

XGBoost (Chen and Guestrin, 2016) is gradient boosting with a regularised objective Σ l(yi, ŷi) + Σk [γTk + ½λ‖wk‖²], where T is the number of leaves and w the leaf weights. Each tree is fitted with a second-order Taylor expansion of the loss using gradients gi and hessians hi.

  • Optimal leaf weight: w* = −G / (H + λ). Split gain: ½[GL²/(HL+λ) + GR²/(HR+λ) − G²/(H+λ)] − γ. A split is kept only if the gain is positive, which is pre-pruning by γ.
  • min_child_weight is the minimum hessian sum in a child. For squared loss h = 1, so it is just a minimum number of dots.
  • This demo uses exact greedy split finding on one feature. Real XGBoost adds column and row subsampling, histogram/approximate splits, sparsity-aware default directions, and a fast parallel C++ core. LightGBM is a close cousin with leaf-wise growth and histogram binning.
  • The wild booster is the same code with λ = γ = 0 and min child weight 1, so any difference you see comes from the regulariser alone.

Data: 80 training dots from a wavy curve plus Gaussian noise (σ = 0.15), 400 fresh dots. Error is RMSE; about 0.15 is the noise floor.