Ethics, Safety & Alignment

Why Did It Say That?

A robot says "cat" or "dog". Ask it to show its reasons, change one clue, and catch it cheating with a silly shortcut.

These photos are pretend measurements. Slide the clues around and watch the bars. Which clue pushes hardest?

Edit a photo

...
catdog

Why? Each clue's push for this photo

Green pushes to "dog", red pushes to "cat", compared with an average photo.

Challenges

Poke it: move any clue slider and watch the bars move.
Flip it: change only the wall colour so the robot changes its answer, while the animal stays the same.
Fix it: get 75% or more on New places.
Honest robot: make familiar and new places within 8 points of each other, with both at 70% or more.

What just happened?

For grown-ups: what's really going on

The model is a logistic regression on standardised features. Because it is linear in the log-odds, the local explanation is exact: for photo x, clue j contributes wj (xj − meanj) / sdj, and the contributions plus the bias add up to the log-odds. For non-linear models, methods such as SHAP and LIME approximate this idea.

  • Permutation importance is a global, model-agnostic measure: shuffle one feature on held-out data and record the accuracy drop (averaged over a few shuffles with a fixed seed).
  • The wall colour is a spurious correlation (shortcut learning): 95% aligned with the label in training, but unrelated in new places. This is why distribution shift breaks models that looked fine on the test set.
  • Explanations show what the model uses, not what is causally true. Correlated features can share or steal credit, and an explanation can reassure while the model is still wrong.
  • The two fixes are feature removal and data augmentation or balancing (decorrelating the shortcut from the label).

Further reading: Ribeiro et al., Why Should I Trust You? (LIME, 2016) · Lundberg and Lee, A Unified Approach to Interpreting Model Predictions (2017) · Geirhos et al., Shortcut Learning in Deep Neural Networks (2020).