Time Series & Forecasting

Spot the Odd One

A sensor reports a number every minute, and sometimes something goes wrong. Set how picky the alarm is: too jumpy cries wolf, too relaxed misses real trouble.

The shaded band is what I expect. Anything outside it rings the alarm. Slide Alarm line and watch the red circles.

Sensor readings

readingexpected bandalarmreal problem (revealed)

The real problems are hidden until you press the button, but your score is always graded against them.

Challenges

Quiet alarm: get 100% alarms right while still ringing at least 5 times.
Catch them all: catch 90% or more of the real problems.
Balanced: get the balance score to 65% or more.
Slow drift: catch the sensor that slowly goes wrong (the green ramp, near time 240 to 275).

What just happened?

For grown-ups: what's really going on

This is a rolling z-score detector: zt = (xt − μt) / σt, where μ and σ are the mean and standard deviation of the previous w readings (never including the current one, so there is no peeking). Alarm if |zt| > threshold. The first w readings are a warm-up with no alarms.

  • Precision = alarms that were real problems ÷ all alarms. Recall = real problems that got at least one alarm ÷ all real problems. F1 is their harmonic mean. Raising the threshold generally raises precision and lowers recall.
  • For pure Gaussian noise, a threshold of 3 flags about 0.3% of normal points, a threshold of 2 about 5%, which at one reading per minute is a lot of false alarms.
  • The hidden labels are 8 big spikes, 4 subtle spikes and one slow drift (points at least 3.5 off). Recall counts each of these 13 events once. Precision counts individual alarms.
  • A rolling baseline adapts to concept drift, so a slow ramp can look normal. Real systems also use seasonal baselines, robust statistics (median and MAD) or learned models such as isolation forests and autoencoders.
  • Anomaly labels are rare in practice, so people often tune the threshold to a tolerable alarm rate rather than to F1.

Further reading: Chandola, Banerjee and Kumar (2009), Anomaly Detection: A Survey.