Data & Features

Who Did You Ask?

The robot learns which people will like a new skate park, but only from the people you ask. Ask just one neighbourhood and see who it gets wrong.

I only know the people you asked. Right now you asked just Hilltop, because it was close by. How will I do in the rest of town?

The town

likes the skate parkdoes notring = you asked them

Faint dots are everyone in town. The colour painted behind them is what the robot would answer there.

Challenges

See the bias: ask only one neighbourhood and find the whole-town score under 65% with one place under 55%.
Fair crowd: get every neighbourhood to 70% or more.
Smart, not big: get every neighbourhood to 65% or more while asking 100 people or fewer.
Big but biased: ask 100 or more people from one neighbourhood only and see the town still stuck under 65%.

What just happened?

For grown-ups: what's really going on

The town has four neighbourhoods of 200 people each. Within each, whether a person likes the skate park depends on which side of a line (through that neighbourhood's centre, at a different angle each) they are on, plus 5% random label noise. The learner is k-nearest neighbours (k = 5) and it is scored on everyone who was not asked.

  • This is sampling bias (selection bias): the training distribution differs from the deployment distribution. A convenience sample is the typical cause. The cure is a probability sample, for example stratified sampling with equal or proportional quotas per group.
  • Formally the trouble is covariate shift (and here also differing P(y|x) per group): the model is confident only where it has data and is arbitrary where it has none. More data from the same region lowers variance there but cannot fix coverage.
  • Reporting only an overall score hides subgroup performance gaps. Fairness audits compare accuracy or error rates per group.
  • Sample order within each neighbourhood is a fixed seeded shuffle, so "ask 20" is always the same 20 people and results are deterministic.
  • Related ideas: importance weighting, active learning (ask where the model is unsure), data documentation such as datasheets for datasets.