Ensembles & Boosting

Bagging vs Boosting

Same dots, same wrong labels, two team styles. Turn up the noise and see which team keeps its cool.

Slide wrong labels up. Watch the lines: the purple team (boosting) can climb and then slump, while the green team (bagging) stays steady.

Score on fresh dots, round by round

one treebaggingboostingboosting on its own training dots

Fresh dots have their true labels, so this shows how well each team learned the real pattern, not the mistakes.

Challenges

Noise test: at 25% wrong labels or more, find a round where boosting scores lower than bagging.
Boosting slump: find a noise level where boosting’s best score is at least 4 points above its score at round 80.
Calm bagging: with 20% or more wrong labels, bagging at round 80 is within 3 points of its own best.
Clean data: set wrong labels to 0% and see boosting do about as well as bagging (within 3 points).

What just happened?

How it works

  1. Make training dots with some labels flipped, and fresh dots with true labels.
  2. Bagging: each round adds one deep tree trained on a bootstrap sample. Prediction is a majority vote.
  3. Boosting: each round adds one depth-3 tree trained on re-weighted dots. Prediction is the sign of the weighted sum.
  4. Score both on the fresh dots after every round.
For grown-ups: what's really going on

Bagging reduces variance by averaging low-bias, high-variance learners (here depth-8 CART trees). Boosting reduces bias by fitting an additive model to the weighted errors (discrete AdaBoost with depth-3 trees).

  • AdaBoost minimises exponential loss exp(−y F(x)), which grows without bound on a mislabelled point, so noisy labels get ever-larger weight. Bagging only resamples; no single dot can dominate.
  • Boosting therefore tends to overfit label noise when run for many rounds without regularisation. Remedies: fewer rounds (early stopping on a validation set), a learning rate (shrinkage), shallower trees, or more robust losses such as the logistic loss used in gradient boosting.
  • Real libraries (XGBoost, LightGBM) boost with such regularisers and often beat bagging on clean tabular data; the point here is the trade-off, not a winner.
  • Training: 150 half-moon dots, label flips at the chosen rate. Test: 500 fresh dots scored against their true class. One seed is noisy, so press New data a few times.