No Peeking, Snacks!
This is the idea behind CatBoost. Trees need numbers, but "Cookie" and "Celery" are names. A common trick swaps each name for "how often kids liked it". Done carelessly, that number peeks at the answer. CatBoost's fix: each row only looks at the rows that came before it.
Two boosters learn whether a kid likes a snack, from its sweetness and its name. One peeks, one does not. Press Train 10 rounds and compare their scores on new kids.
Rounds: 0
Left to right: how sweet. Bottom to top: the number that stands in for the snack's name. With peeking, rare snacks jump to the very top or bottom, right next to their own answer.
Challenges
What just happened?
- A snack name is a category, not a number. One neat trick: replace it with how often kids liked that snack.
- But if Fig was only eaten once, and that kid loved it, Fig's number is "loved": the row's own answer sneaks into its clue.
- The peeking booster learns to trust that clue. It looks brilliant on the kids it trained on, then flops on new kids.
- The no-peeking way lines the kids up in a random order. Each kid only sees how the kids before them liked the snack, never their own answer.
How it works
- Shuffle the training kids into a random line.
- Walk down the line. For each kid, count how many earlier kids had the same snack and how many liked it.
- Snack number = (liked so far + a little of the overall average) ÷ (seen so far + a little). The first kid to try a snack just gets the average.
- Boost trees on sweetness and that snack number.
- For new kids, use all the training kids' counts. Their own answers were never in it.
For grown-ups: what's really going on
CatBoost (Prokhorenkova et al., 2018) handles categorical features with ordered target statistics. For a random permutation σ, example k gets x̂k = (Σj: σ(j) < σ(k), xj = xk yj + a·p) / (#{j: σ(j) < σ(k), xj = xk} + a), with prior p (the mean label) and strength a.
- Greedy target encoding uses each example's own label, so
E[x̂k | yk]differs between train and test. This target leakage causes a prediction shift, which hurts most for rare categories. - CatBoost also uses ordered boosting: the residuals for example k come from a model that never saw k, using the same permutation idea. It averages several permutations and grows symmetric (oblivious) trees. This demo shows only the ordered statistics, with ordinary boosting on top.
- Other fixes for the leak: K-fold target encoding, or leave-one-out encoding plus noise. One-hot encoding avoids leakage but explodes with many categories.
Data: 300 training rows, 1500 fresh rows. 40 snacks with skewed popularity; P(like) = σ(6·(sweet − 0.5) + bonus), where only 6 snacks have a real bonus. Boosting: log loss, depth-2 trees, learning rate 0.2, prior strength a = 1.