Generative Models

Spot the Stranger

A network that only practised on one kind of picture can't rebuild anything else. Its mistakes give away the strangers.

I learned to rebuild only one kind of picture. Strangers look wrong to me, so my rebuild is bad. Slide the alarm line and catch them all!

The only kind it learns (normal)

Rebuild mistakes

normal picturestrangeralarm line

Top: the picture. Bottom: the rebuild. A red frame means the alarm rang.

Challenges

Jumpy alarm: set the line so low that every stranger is caught (100% recall) but precision falls under 70%.
Sleepy alarm: set the line so high that the alarm only rings for real strangers (100% precision) yet it catches less than 60% of them.
Just right: reach a balance of 90% or more.
Different job: pick another normal kind, wait for it to learn, and reach a balance of 90% again.

What just happened?

For grown-ups: what's really going on

This is anomaly detection (or novelty detection) with an autoencoder (64 → 24 → 3 → 24 → 64). It is trained on 150 "normal" pictures only; the 3-number bottleneck leaves room for the normal shapes but not for the others.

  • Anomaly score = reconstruction error (mean squared error). A picture is flagged when its score is above the threshold.
  • Precision = TP / (TP + FP). Recall = TP / (TP + FN). F1 = 2TP / (2TP + FP + FN). Moving the threshold trades one for the other, tracing a precision-recall curve.
  • The test set is 30 normal and 30 stranger pictures, so one error moves the numbers by 3 percentage points. Real fraud or fault data has far fewer strangers, which makes precision much harder.
  • Autoencoders can be too good: a big bottleneck rebuilds strangers well too, and the detector stops working. Try the previous lessons' idea in reverse.

Further reading: Chandola, Banerjee and Kumar, Anomaly Detection: A Survey (2009); Sakurada and Yairi (2014), Anomaly detection using autoencoders.