Why Did It Say That?
A robot says "cat" or "dog". Ask it to show its reasons, change one clue, and catch it cheating with a silly shortcut.
These photos are pretend measurements. Slide the clues around and watch the bars. Which clue pushes hardest?
Edit a photo
Why? Each clue's push for this photo
Green pushes to "dog", red pushes to "cat", compared with an average photo.
Challenges
What just happened?
- The robot showed its reasons: each clue pushed the answer towards cat or dog.
- In the practice photos most dogs happened to stand in front of a blue wall. The robot found that easier than looking at ears and noses.
- On familiar photos it looked brilliant. On new places it was nearly guessing. Only the explanation exposed why.
- Fixing it meant removing the shortcut or giving it better practice data. Explanations help people spot that kind of problem.
For grown-ups: what's really going on
The model is a logistic regression on standardised features. Because it is linear in the log-odds, the local explanation is exact: for photo x, clue j contributes wj (xj − meanj) / sdj, and the contributions plus the bias add up to the log-odds. For non-linear models, methods such as SHAP and LIME approximate this idea.
- Permutation importance is a global, model-agnostic measure: shuffle one feature on held-out data and record the accuracy drop (averaged over a few shuffles with a fixed seed).
- The wall colour is a spurious correlation (shortcut learning): 95% aligned with the label in training, but unrelated in new places. This is why distribution shift breaks models that looked fine on the test set.
- Explanations show what the model uses, not what is causally true. Correlated features can share or steal credit, and an explanation can reassure while the model is still wrong.
- The two fixes are feature removal and data augmentation or balancing (decorrelating the shortcut from the label).
Further reading: Ribeiro et al., Why Should I Trust You? (LIME, 2016) · Lundberg and Lee, A Unified Approach to Interpreting Model Predictions (2017) · Geirhos et al., Shortcut Learning in Deep Neural Networks (2020).