Where AI fails
AI can be right a lot and still be confidently wrong. Here are four tiny AIs. Train each one, guess whether it will get things right, then find out!
When should I trust it?
Each habit lights up when you finish a challenge in that stage.
Is this like its practice?
Ask: has the AI seen things like this before? Far from home, it is just guessing, however sure it sounds.What is it really looking at?
Check it works when the background or the extra clues change. It may have found a shortcut.Does it survive a wiggle?
Try small changes. If a tiny nudge flips the answer, do not lean on it for big decisions.Can I double-check it?
Fluent does not mean true. Look facts up in a real source before you share them.How it works
Learn from examples
Each AI only knows the examples it was shown. Nothing else.
Score well on the test
A test made of the same kind of examples says "great!". That is the easy test.
Meet something different
A new kind of input, a changed background, a tiny nudge, or a question about something unread.
Always an answer
The AI has to say something. It has no built-in "I don't know", so it sounds just as sure.
Check, then trust
Test the strange cases, look for shortcuts, and ask a person or a book to double-check.
What just happened?
- Every AI scored high on its normal test and still failed once the situation changed. A high score only tells you about the kind of examples it was tested on.
- Far from its practice, the first AI stayed very sure. Being sure is not the same as being right.
- The second AI found a shortcut (snow) instead of the real clue (the shape), so it fell apart when the snow moved.
- A tiny, carefully aimed nudge flipped the third AI's answer, but a random nudge of the same size did not.
- The fourth AI wrote a smooth sentence with a made-up fact. Always double-check.
For grown-ups: what's really going on
All four models are small and trained live in your browser: two logistic regressions on 2 and 64 features (full-batch gradient descent), a third on bar pictures, and a word-level trigram model with back-off to bigram. They are transparent stand-ins for failure modes seen in real systems, not claims about any specific product.
- Distribution shift / out-of-distribution inputs: the linear model is trained on animals clustered around two points. Test accuracy on the same distribution is about 100%. Outside it, the output depends only on which side of the line a point falls, and confidence grows with distance from the boundary, so far-away points get near-100% confidence. This is a calibration failure: softmax and sigmoid outputs are not honest uncertainty about novel inputs.
- Shortcut learning / spurious correlation: in training, a 4-pixel "snow" patch perfectly predicts the label while the real shape is noisy. The model leans on the patch: about 100% accuracy when the cue matches, about 0% when it is flipped (seeds 1 to 3). Training with the cue randomised gives about 95% in both cases. Known real examples include classifiers using backgrounds or watermarks.
- Adversarial examples: the nudge is the fast gradient sign method: each pixel moves by epsilon in the direction of the sign of the weight that lowers the current class. In 64 dimensions many tiny changes add up (the margin shrinks by epsilon times the sum of absolute weights). Median flip size is about 0.07 of the pixel range, and at epsilon 0.1 about 85% of pictures flip, against about 0 to 4% for random signs of the same size.
- Hallucination: a language model is trained to produce a plausible next token, not a true one. Here, for an unseen country, back-off to the previous word's statistics produces a fluent, wrong city. Probabilities are shown honestly (about 13% for the first word) but the output sentence carries no marker of that uncertainty.
- Mitigation: test on shifted and adversarial data, not just held-out data from the same distribution; use uncertainty and out-of-distribution detection (a novelty alarm like the one in stage 1), calibration, data augmentation that breaks shortcuts, retrieval or citation checks for facts, and human review for high-stakes decisions.
- Further reading: Shortcut Learning in Deep Neural Networks (Geirhos et al.), Explaining and Harnessing Adversarial Examples (Goodfellow et al.), On Calibration of Modern Neural Networks (Guo et al.).