Agents & Tools

Pick the Right Tool

A helper with many tools must choose one for each request. Tell Robo the right tool for a few requests and watch it learn to choose.

Each request needs a calculator, a clock, a lookup table, or no tool. Tap the right one for a few. The router that learns is real, but the tools are tiny toys.

Requests

Tap the tool this request really needs. The small code on the right is Robo's current pick and how sure it is.

Challenges

See a wrong tool fail: ask your own request and watch the tool error out, or fix a wrong pick.
Get 60% or more right with only 8 or fewer labelled requests.
Reach 85% right on the secret test.
Ask your own request, then tap the tool it should have used to teach Robo.
Reach 95% right.

What just happened?

For grown-ups: what's really going on

The router is a multinomial Naive Bayes text classifier over a bag of words: P(tool | words) ∝ P(tool) · ∏ P(word | tool) with add-one smoothing. Numbers become a <num> token and arithmetic signs become <op>, so it generalises to numbers it never saw.

  • This is a small version of tool selection (also called routing or function calling). Real systems let a language model choose a tool and write its arguments; here a learned classifier chooses and simple code extracts the numbers.
  • The tools are toy implementations: a calculator that reads two numbers, a fixed clock, and a tiny lookup table.
  • Accuracy is measured on held-out requests. Failures come in two kinds: the tool errors (loud, easy to notice) or it answers the wrong question (silent, worse).
  • Confidence is the posterior probability. Naive Bayes tends to be over-confident because it assumes words are independent.

Further reading: Schick et al., Toolformer (2023) · Jurafsky and Martin, Speech and Language Processing, ch. 4.