Tidy the Mess
A robot guesses a dog's weight from its height, but the table is full of blanks, typos and silly numbers. Clean it up and watch the score.
Here is the dog table, straight from a messy notebook. I will learn from it exactly as it is. Press a clean-up button and see what happens to my score!
The dog notebook
| Dog | Height (cm) | Weight (kg) |
|---|
yellow = blank, red = wild value. Green text shows what the robot sees after your clean-up.
good dogwas blankwildrobot's line
Challenges
Garbage in: leave everything messy and see the score fall under 30%.
Clean it up: reach a score of 85% or more.
Waste not: reach 85% while still keeping at least 50 of the 60 rows.
Mean or median? Try both fill options (with any wild-value setting).
What just happened?
- The robot learns only from the table it is given. A blank read as 0 or a height of 450 cm pulls its line far away from the truth.
- The test dogs were clean and hidden, so the score shows how the line works on real dogs.
- Dropping bad rows works but throws away data. Filling blanks keeps rows, and predicting a blank from the other column is usually the smartest fill.
- The mean is dragged by wild values, the median is not. That is why outliers and fills are best handled together.
- Clipping squeezes wild values into the normal range, but a typo is still a wrong number. Sometimes dropping it is more honest.
For grown-ups: what's really going on
The model is ordinary least squares simple linear regression: weight = a · height + b with a = cov(x,y)/var(x). Score is R² = 1 − SSres/SStot on held-out rows, clamped at 0, and the typical miss is the mean absolute error.
- Wild values are found with Tukey fences: values beyond
Q1 − 1.5·IQRorQ3 + 1.5·IQR. Clipping to the fences is called winsorising. - The typos here are decimal slips (a height ×10 or a weight ÷10); outliers are weights ×3 to ×6. Both are injected with a seeded random generator, so the same notebook is deterministic.
- Mean imputation shrinks variance and pulls toward the centre; median imputation is robust to outliers; regression imputation fills a blank from the other column, which keeps the relationship but makes data look tidier than it is.
- Leaving blanks as 0 is a classic silent bug: it is a legal number, so nothing crashes, but it is wrong.
- Dropping rows is fine when blanks are random (missing completely at random). If blanks are linked to the answer, dropping them biases the model.