Spot the Stranger
A network that only practised on one kind of picture can't rebuild anything else. Its mistakes give away the strangers.
I learned to rebuild only one kind of picture. Strangers look wrong to me, so my rebuild is bad. Slide the alarm line and catch them all!
The only kind it learns (normal)
Rebuild mistakes
normal picturestrangeralarm line
Top: the picture. Bottom: the rebuild. A red frame means the alarm rang.
Challenges
Jumpy alarm: set the line so low that every stranger is caught (100% recall) but precision falls under 70%.
Sleepy alarm: set the line so high that the alarm only rings for real strangers (100% precision) yet it catches less than 60% of them.
Just right: reach a balance of 90% or more.
Different job: pick another normal kind, wait for it to learn, and reach a balance of 90% again.
What just happened?
- The network practised on only one kind of picture, so it learned that kind's shapes and nothing else. Anything different is rebuilt badly.
- The rebuild error is an "oddness score". No stranger was ever shown to it and nobody labelled the strangers during training.
- The alarm line is a trade-off. Low: you catch every stranger but cry wolf. High: no false alarms but strangers slip by. Precision and recall measure the two sides.
- Some strangers look a bit like normal pictures (a ring is a square with a hole), so they are harder to catch.
For grown-ups: what's really going on
This is anomaly detection (or novelty detection) with an autoencoder (64 → 24 → 3 → 24 → 64). It is trained on 150 "normal" pictures only; the 3-number bottleneck leaves room for the normal shapes but not for the others.
- Anomaly score = reconstruction error (mean squared error). A picture is flagged when its score is above the threshold.
- Precision = TP / (TP + FP). Recall = TP / (TP + FN). F1 = 2TP / (2TP + FP + FN). Moving the threshold trades one for the other, tracing a precision-recall curve.
- The test set is 30 normal and 30 stranger pictures, so one error moves the numbers by 3 percentage points. Real fraud or fault data has far fewer strangers, which makes precision much harder.
- Autoencoders can be too good: a big bottleneck rebuilds strangers well too, and the detector stops working. Try the previous lessons' idea in reverse.
Further reading: Chandola, Banerjee and Kumar, Anomaly Detection: A Survey (2009); Sakurada and Yairi (2014), Anomaly detection using autoencoders.