Ensembles & Boosting

Wisdom of the Crowd

One person guessing jellybeans is usually wrong. A whole crowd averaged together is often amazingly close.

The jar has a secret number of jellybeans. Everyone guesses a bit too high or too low. Slide the crowd size up and watch the average settle on the truth!

Everybody's guesses

Green line = the true number. Purple arrow = the crowd's average. Each dot is one person.

Challenges

Big crowd: use 100 or more people, and make the crowd at least twice as close as a typical person.
Fair fight: ask 5 random people. The crowd should beat at least 4 of them.
Fooled together: set the shared bias to 20% or more, use 100+ people, and see the crowd stay wrong by at least 10%.

What just happened?

How it works

  1. Everyone guesses the jar. Each guess is the truth, plus a shared lean, plus personal noise.
  2. We average the first N guesses.
  3. We compare that average with the truth, and with how far off a typical single person is.
  4. The chart repeats the whole experiment 300 times with fake crowds, so the line is a fair average and not one lucky jar.
For grown-ups: what's really going on

Each guess is modelled as gi = T(1 + b + σ·εi) with εi ~ N(0,1), σ = 0.3 and bias b shared by everyone. The crowd mean has bias T·b and variance (Tσ)²/n, so its typical error falls like 1/√n until it hits the floor set by the bias.

  • This is the idea behind variance reduction by averaging, used by bagging and random forests: average many noisy but roughly unbiased models.
  • It needs the errors to be independent. If errors are correlated with correlation ρ, the variance floors at ρσ² instead of going to zero. A shared bias is the extreme case.
  • The jellybean story is the classic Galton ox-weight example (1907) and is why ensembles work: diversity matters as much as numbers.

The chart is a Monte Carlo estimate (300 simulated crowds). The jar and crowd come from a seeded random generator, so a seed always gives the same crowd.