Supervised Learning

Spam Filter

Junk messages use sneaky words. Sort a few messages by hand and watch a robot work out which words are the suspicious ones.

Here is a pile of messages. Tap Spam or Fine on a few of them. I will tell you which words you were looking at, and start learning too!

The message pile

The bar under each message is the robot's guess, how likely it is spam. Tap a message to peek inside the robot's head.

Challenges

Teach it well: get 90% right on the secret test messages.
Tiny teacher: get 75% or more with only 2 to 5 labelled messages. Pick them wisely!
Sneaky spammer: after training, switch it on. Spam with fake spellings fools the robot!
Fight back: label the new sneaky messages (or press Robot) until it catches 80% again.
New topic: normal chat that uses words like FREE. Watch the false alarms climb to 3 or more.
Fix the false alarms: teach it the new messages are Fine until there are 2 or fewer.

What just happened?

For grown-ups: what's really going on

This is a multinomial Naive Bayes classifier on a bag of words: each message becomes a list of lowercase word counts and word order is ignored. "Naive" because it assumes words are independent given the class: P(spam | words) ∝ P(spam) · ∏ P(word | spam).

  • It is trained incrementally: each label just updates counts, so it is exact and instant. Un-marking a message subtracts its counts.
  • Laplace (add-one) smoothing: P(w | c) = (count(w, c) + 1) / (total words in c + V), so a word never seen in one class does not zero the whole product. V is the vocabulary size so far.
  • We work in log-odds: log P(spam)/P(fine) + Σ log P(w|spam)/P(w|fine). Sums are stable and each word's term is its colour-coded score. Probability = 1 / (1 + e−total). Words never seen are skipped.
  • Accuracy is measured on held-out messages. Misspellings and distribution shift (a new topic) break the word statistics; retraining on fresh labels repairs them.
  • Accuracy hides the kinds of error. Precision = of messages flagged spam, how many really were (false alarms hurt this). Recall = of real spam, how many were caught (misses hurt this). The next lesson digs into both.

Further reading: Jurafsky and Martin, Speech and Language Processing, ch. 4 (Naive Bayes) · Paul Graham, A Plan for Spam (2002).