Right or Wrong?
A dragon detector that is 99% accurate can still be useless. Scan 1000 villagers, count the false alarms and the misses, and find out why.
Each villager has a dragon score. You are the detector: ring the alarm for every score above your line. Drag the alarm line!
Red = dragons, grey = ordinary villagers (each group drawn at its own height so you can see the dragons). Star = best line for this job.
Watch it learn
A tiny robot (logistic regression) studies the same villagers and learns its own dragon score by gradient descent. Watch accuracy shoot up while recall lags behind.
Challenges
What just happened?
- Four boxes tell the whole story: right alarms, false alarms, misses and right quiet.
- When dragons are rare, saying "no" to everyone is almost always right, so accuracy hides the problem.
- Precision asks "of the alarms, how many were real?". Recall asks "of the real ones, how many did we catch?".
- Moving the line trades one for the other. Which mistake is worse decides where the line goes.
- The robot learns on lopsided data and plays it safe. Giving rare examples more weight makes it catch more.
For grown-ups: what's really going on
The four boxes are the confusion matrix: TP, FP, FN, TN. Accuracy = (TP+TN)/N. Precision = TP/(TP+FP). Recall (sensitivity) = TP/(TP+FN). F1 = 2PR/(P+R), the harmonic mean, shown here as 0 for the current line.
- The accuracy paradox: with 1% positives, a classifier that always predicts negative scores 99% accuracy and 0 recall. On imbalanced data, report precision, recall, or F1 instead.
- The precision/recall trade-off: raising the decision threshold cuts false positives but adds false negatives. Sweeping it traces the ROC curve (TPR vs FPR) and the precision-recall curve; PR curves are more informative under heavy imbalance because precision depends on prevalence.
- The star uses a cost-sensitive choice: the threshold minimising
cost_FN·FN + cost_FP·FPover the scores. - The learner is logistic regression with full-batch gradient descent on standardised features and weighted log-loss. Class weights multiply the loss of the rare class; this shifts predicted probabilities up, similar to lowering the threshold. Other remedies: resampling (SMOTE, undersampling), threshold tuning, better features.
- Caveat: metrics here are on the same villagers used for training (only 3 parameters, so little overfitting). A real project scores on held-out data.
Further reading: scikit-learn user guide, "Metrics and scoring" · Saito and Rehmsmeier (2015) on precision-recall plots.