Agents & Tools

A Small Backpack

A chat helper can only hold so many words in its backpack. Make the backpack smaller and see which memory trick still remembers your facts.

Drag the backpack size. Three memory tricks try to answer questions about the chat. The reader is a simple toy, not a real language model.

The chat

word for wordsquashed summaryfetched by searchforgottena fact to remember

Questions about the chat (tap one)

Challenges

Shrink the backpack until keep-last remembers 2 or fewer of the facts.
Find a size where search memory gets 7 or more while keep-last gets 3 or fewer.
Find a size where the summary beats keep-last by 2 or more.
Make the backpack 40 words or smaller and still get 8 out of 8 with search memory.
Make the backpack big enough that all three tricks get 8 out of 8.

What just happened?

For grown-ups: what's really going on

A language model can only read a limited context window (measured in tokens; here, words). Anything outside it is invisible. An agent needs an external memory policy to decide what goes back in.

  • Keep-last-N (a sliding window): newest turns that fit the budget.
  • Summarise: half the budget is recent turns; older turns are compressed by removing stopwords and ranked by a hand-written importance score (digits and capitalised names), keeping the best that fit. A real system asks a model to write the summary, which is also lossy.
  • Retrieval (the idea behind RAG): one third recent turns, the rest the top-3 older turns by TF-IDF cosine similarity to the question. tfidf(w) = tf · (ln((N+1)/(df+1)) + 1), so rare words count for more than common ones.
  • Toy reader: a keyword and value matcher stands in for the language model that would read the context. Scores show whether the right line reached the context, which is the thing memory controls.
  • Metric: number of the 8 recall questions answered correctly. Facts lean towards the start of the chat, so small windows hurt.

Further reading: Lewis et al., Retrieval-Augmented Generation (2020) · Liu et al., Lost in the Middle (2023).