Data & Features

Tidy the Mess

A robot guesses a dog's weight from its height, but the table is full of blanks, typos and silly numbers. Clean it up and watch the score.

Here is the dog table, straight from a messy notebook. I will learn from it exactly as it is. Press a clean-up button and see what happens to my score!

The dog notebook

DogHeight (cm)Weight (kg)

yellow = blank, red = wild value. Green text shows what the robot sees after your clean-up.

good dogwas blankwildrobot's line

Challenges

Garbage in: leave everything messy and see the score fall under 30%.
Clean it up: reach a score of 85% or more.
Waste not: reach 85% while still keeping at least 50 of the 60 rows.
Mean or median? Try both fill options (with any wild-value setting).

What just happened?

For grown-ups: what's really going on

The model is ordinary least squares simple linear regression: weight = a · height + b with a = cov(x,y)/var(x). Score is R² = 1 − SSres/SStot on held-out rows, clamped at 0, and the typical miss is the mean absolute error.

  • Wild values are found with Tukey fences: values beyond Q1 − 1.5·IQR or Q3 + 1.5·IQR. Clipping to the fences is called winsorising.
  • The typos here are decimal slips (a height ×10 or a weight ÷10); outliers are weights ×3 to ×6. Both are injected with a seeded random generator, so the same notebook is deterministic.
  • Mean imputation shrinks variance and pulls toward the centre; median imputation is robust to outliers; regression imputation fills a blank from the other column, which keeps the relationship but makes data look tidier than it is.
  • Leaving blanks as 0 is a classic silent bug: it is a legal number, so nothing crashes, but it is wrong.
  • Dropping rows is fine when blanks are random (missing completely at random). If blanks are linked to the answer, dropping them biases the model.