Bagging vs Boosting
Same dots, same wrong labels, two team styles. Turn up the noise and see which team keeps its cool.
Slide wrong labels up. Watch the lines: the purple team (boosting) can climb and then slump, while the green team (bagging) stays steady.
Score on fresh dots, round by round
one treebaggingboostingboosting on its own training dots
Fresh dots have their true labels, so this shows how well each team learned the real pattern, not the mistakes.
Challenges
Noise test: at 25% wrong labels or more, find a round where boosting scores lower than bagging.
Boosting slump: find a noise level where boosting’s best score is at least 4 points above its score at round 80.
Calm bagging: with 20% or more wrong labels, bagging at round 80 is within 3 points of its own best.
Clean data: set wrong labels to 0% and see boosting do about as well as bagging (within 3 points).
What just happened?
- Bagging trains many deep trees on shuffled copies of the data. They each memorise noise differently, and voting cancels it out. More trees never hurt much.
- Boosting trains small trees one after another and makes the hard dots count more each round. A wrongly labelled dot is always hard, so boosting keeps putting more and more attention on it.
- On clean data boosting is great. With many wrong labels it starts bending the boundary around the mistakes, and fresh-dot accuracy falls.
- The amber line is boosting on its own training dots: it keeps going up toward 100% by learning the noise, while the purple line on fresh dots does not.
How it works
- Make training dots with some labels flipped, and fresh dots with true labels.
- Bagging: each round adds one deep tree trained on a bootstrap sample. Prediction is a majority vote.
- Boosting: each round adds one depth-3 tree trained on re-weighted dots. Prediction is the sign of the weighted sum.
- Score both on the fresh dots after every round.
For grown-ups: what's really going on
Bagging reduces variance by averaging low-bias, high-variance learners (here depth-8 CART trees). Boosting reduces bias by fitting an additive model to the weighted errors (discrete AdaBoost with depth-3 trees).
- AdaBoost minimises exponential loss
exp(−y F(x)), which grows without bound on a mislabelled point, so noisy labels get ever-larger weight. Bagging only resamples; no single dot can dominate. - Boosting therefore tends to overfit label noise when run for many rounds without regularisation. Remedies: fewer rounds (early stopping on a validation set), a learning rate (shrinkage), shallower trees, or more robust losses such as the logistic loss used in gradient boosting.
- Real libraries (XGBoost, LightGBM) boost with such regularisers and often beat bagging on clean tabular data; the point here is the trade-off, not a winner.
- Training: 150 half-moon dots, label flips at the chosen rate. Test: 500 fresh dots scored against their true class. One seed is noisy, so press New data a few times.