Tricked by Words
A pretend assistant reads an email and obeys any order it finds, even one a sneaky writer hid inside. Try four defences and score them.
Pick a sneaky document and press Run. Then switch on defences and watch the score at the right.
The user asks
What a person sees in the document
Challenges
Get tricked: with no defences, run a sneaky document and see the assistant obey it.
Fake marker: switch on only "Mark the text as data", and notice the fake end marker trick still gets through.
Find a hole: use only the allow-list and see a trick that uses an allowed action (changing who gets the message).
Tired user: use only "Ask before acting", slide the tired user to 30% or more, and watch the score get worse.
Layers: stop all 12 tricks while all 4 normal jobs still work.
What just happened?
- The toy assistant could not tell the user's orders from orders written inside the email. It simply obeyed any order-like sentence. That is called prompt injection.
- Each defence stopped some tricks and missed others: marking text as data, an allow-list of actions, and asking before risky actions.
- Layering them worked much better than any one alone. A tired user who clicks Yes without reading weakens "ask first".
- This is only a toy with 12 fixed tricks. Real attackers invent new ones, so real systems also limit what the assistant is allowed to do.
For grown-ups: what's really going on
There is no language model here. The "assistant" is a small rule-based parser that treats every order-like sentence in its whole input as an instruction, which reproduces the core flaw of prompt injection: instructions and data share one text channel. Everything runs locally, and every action is only a string.
- Data/instruction separation (delimiters, or structured roles): the assistant is told that text inside the data tags is not to be obeyed. Here, the parser ignores it unless the attacker closes the tag early (the fake end marker), so sanitising the markers matters. Real models also obey marked data some of the time, so the toy leaks with a fixed 30% probability.
- Allow-list / least privilege: each task may only use certain actions. It cannot stop an attack that uses an allowed action in a harmful way (sending to a different address).
- Human confirmation for risky actions the user did not request. It is weakened by confirmation fatigue, simulated by the tired-user slider (seeded, so results repeat).
- An indirect injection arrives through content the assistant reads (emails, web pages, files), not from the user. Hidden text (white on white, tiny print) is invisible to people but not to the assistant.
- The attack success rate is measured on a fixed list, so it is a regression test, not a guarantee. Real defence is defence in depth: least privilege, isolating untrusted content, output checks, logging, and human review for high-impact actions.
Further reading: Greshake et al., Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023) · OWASP Top 10 for LLM Applications.