Who Did You Ask?
The robot learns which people will like a new skate park, but only from the people you ask. Ask just one neighbourhood and see who it gets wrong.
I only know the people you asked. Right now you asked just Hilltop, because it was close by. How will I do in the rest of town?
The town
likes the skate parkdoes notring = you asked them
Faint dots are everyone in town. The colour painted behind them is what the robot would answer there.
Challenges
See the bias: ask only one neighbourhood and find the whole-town score under 65% with one place under 55%.
Fair crowd: get every neighbourhood to 70% or more.
Smart, not big: get every neighbourhood to 65% or more while asking 100 people or fewer.
Big but biased: ask 100 or more people from one neighbourhood only and see the town still stuck under 65%.
What just happened?
- The robot can only copy patterns from the people it was shown. People from a place nobody asked look like strangers to it, so it guesses.
- Asking more people from the same place made the Hilltop score better but did almost nothing for the rest of town. A bigger biased sample is still biased.
- A varied sample, with some people from every neighbourhood, fixed it with fewer people.
- The score for the whole town can look fine while one group is badly treated, so always check each group separately.
- Real examples: a face tool trained mostly on one skin tone, or a voice tool trained on one accent. The model is not mean, the data was narrow.
For grown-ups: what's really going on
The town has four neighbourhoods of 200 people each. Within each, whether a person likes the skate park depends on which side of a line (through that neighbourhood's centre, at a different angle each) they are on, plus 5% random label noise. The learner is k-nearest neighbours (k = 5) and it is scored on everyone who was not asked.
- This is sampling bias (selection bias): the training distribution differs from the deployment distribution. A convenience sample is the typical cause. The cure is a probability sample, for example stratified sampling with equal or proportional quotas per group.
- Formally the trouble is covariate shift (and here also differing
P(y|x)per group): the model is confident only where it has data and is arbitrary where it has none. More data from the same region lowers variance there but cannot fix coverage. - Reporting only an overall score hides subgroup performance gaps. Fairness audits compare accuracy or error rates per group.
- Sample order within each neighbourhood is a fixed seeded shuffle, so "ask 20" is always the same 20 people and results are deterministic.
- Related ideas: importance weighting, active learning (ask where the model is unsure), data documentation such as datasheets for datasets.