Pick the Right Tool
A helper with many tools must choose one for each request. Tell Robo the right tool for a few requests and watch it learn to choose.
Each request needs a calculator, a clock, a lookup table, or no tool. Tap the right one for a few. The router that learns is real, but the tools are tiny toys.
Requests
Tap the tool this request really needs. The small code on the right is Robo's current pick and how sure it is.
Challenges
What just happened?
- Robo turns each request into a bag of words and counts which words go with which tool.
- Number-like words and signs like + point to the calculator. Words like time and today point to the clock.
- A wrong pick is not just a wrong label: the calculator cannot tell the time, and the clock cannot add. Mistakes cost real answers.
- More labelled examples, and examples that look alike but need different tools, make the router better.
- The router really learns. The four tools are tiny toys, and nothing here is a real language model.
For grown-ups: what's really going on
The router is a multinomial Naive Bayes text classifier over a bag of words: P(tool | words) ∝ P(tool) · ∏ P(word | tool) with add-one smoothing. Numbers become a <num> token and arithmetic signs become <op>, so it generalises to numbers it never saw.
- This is a small version of tool selection (also called routing or function calling). Real systems let a language model choose a tool and write its arguments; here a learned classifier chooses and simple code extracts the numbers.
- The tools are toy implementations: a calculator that reads two numbers, a fixed clock, and a tiny lookup table.
- Accuracy is measured on held-out requests. Failures come in two kinds: the tool errors (loud, easy to notice) or it answers the wrong question (silent, worse).
- Confidence is the posterior probability. Naive Bayes tends to be over-confident because it assumes words are independent.
Further reading: Schick et al., Toolformer (2023) · Jurafsky and Martin, Speech and Language Processing, ch. 4.