Supervised Learning

Right or Wrong?

A dragon detector that is 99% accurate can still be useless. Scan 1000 villagers, count the false alarms and the misses, and find out why.

Each villager has a dragon score. You are the detector: ring the alarm for every score above your line. Drag the alarm line!

a real dragon an ordinary villager

Red = dragons, grey = ordinary villagers (each group drawn at its own height so you can see the dragons). Star = best line for this job.

Watch it learn

A tiny robot (logistic regression) studies the same villagers and learns its own dragon score by gradient descent. Watch accuracy shoot up while recall lags behind.

Round
0
Accuracy
0
Recall
0
Precision
0

accuracyrecall (dragons caught)challenge done

Challenges

Press the Lazy robot: it never rings and is still 95%+ accurate. Check its recall.
Make the rare thing rarer: start above 5% dragons, then slide down to 1% or less. Accuracy rises, the detector does not get better.
Medical test: misses are scary. Choose it and put the line near the star.
Spam filter: false alarms are scary. Choose it and put the line near the star.
Class weights: make a dragon count 25x or more in training so the robot catches at least 60% of dragons.
Out of 1000 villagers. Real dragons are rare.
Makes the robot care more about dragon mistakes while learning.

What just happened?

For grown-ups: what's really going on

The four boxes are the confusion matrix: TP, FP, FN, TN. Accuracy = (TP+TN)/N. Precision = TP/(TP+FP). Recall (sensitivity) = TP/(TP+FN). F1 = 2PR/(P+R), the harmonic mean, shown here as 0 for the current line.

  • The accuracy paradox: with 1% positives, a classifier that always predicts negative scores 99% accuracy and 0 recall. On imbalanced data, report precision, recall, or F1 instead.
  • The precision/recall trade-off: raising the decision threshold cuts false positives but adds false negatives. Sweeping it traces the ROC curve (TPR vs FPR) and the precision-recall curve; PR curves are more informative under heavy imbalance because precision depends on prevalence.
  • The star uses a cost-sensitive choice: the threshold minimising cost_FN·FN + cost_FP·FP over the scores.
  • The learner is logistic regression with full-batch gradient descent on standardised features and weighted log-loss. Class weights multiply the loss of the rare class; this shifts predicted probabilities up, similar to lowering the threshold. Other remedies: resampling (SMOTE, undersampling), threshold tuning, better features.
  • Caveat: metrics here are on the same villagers used for training (only 3 parameters, so little overfitting). A real project scores on held-out data.

Further reading: scikit-learn user guide, "Metrics and scoring" · Saito and Rehmsmeier (2015) on precision-recall plots.