How do you manage agents’ context? RAG, LLM Wiki and GraphRAG

I distinctly remember being little, about 10 or 11, when we lived in Ponta Delgada and I would come home ridiculously excited to watch documentaries on the History Channel and Discovery. Every day, without fail, my head would memorise thousands of details: “Oh, the ancient Greeks had machines in some temples that dispensed holy water in exchange for coins!” It took years and years to build a habit I still have today. Over these past (almost) 36 years, I must have watched tens of thousands of documentaries and videos about science, history, philosophy and other subjects. And the (obvious) truth is that 99.9% of what I heard, saw and read didn’t stick anywhere in my head.

Some facts did stick, because they were as curious as those Greek machines, or because I wrote them down and eventually read them again, or because I internalised them so thoroughly that they are now part of my general knowledge, or because they are associated with other memorable experiences, or simply because the memory is still fresh.

There is an interesting analogy here, though not an equivalence. Foundation models (AI models trained on large volumes of diverse data that can be adapted to different tasks) do not have everything available at every instant either. But we should distinguish knowledge learned during training from active context, the information they receive for a particular model call.

During training, they are exposed to more data than any human could consume in an entire lifetime. Some of that content joins the amalgam of the model’s background knowledge. And, just like humans, models can absorb fake news and false ideas.

In normal, everyday use, models combine that background knowledge with a range of context sources. If I want an AI agent to automate filtering and prioritising my email inbox, it will use the background knowledge it brings, the process knowledge (instructions) I gave it and the data in the emails themselves (knowledge specific to the task). But here we stumble over one of this technology’s biggest limitations: models have a limited context window. Some frontier models have windows in the region of 1 million tokens. On average, and as an approximation for English text, each token corresponds to about four characters. A million tokens, then, is roughly equivalent to one and a half times the text of all three volumes of The Lord of the Rings, in the original English.

Here, I define an agent as a program that uses a foundation model (FM) to make decisions and use tools. Whenever it needs to receive information and make a decision, it calls the model. Information not included in that call will not be available as context, even if it appeared in an earlier call. The industry has been developing ways around this limitation. I refer to the set of practices used to give the agent the context it needs, when it needs it, as context management techniques. One of the underlying ideas is progressive disclosure of information, meaning the way context is supplied over time, alongside tools that let the agent retrieve the information it needs.

As this is all a relatively new field, there is no one-size-fits-all solution. There are several heuristics commonly used in the industry, and one of the most widespread is Retrieval-Augmented Generation (RAG). With this technique, documents are usually divided into chunks according to various criteria, including fixed size, semantic context and document hierarchy. Those chunks are often converted into numerical representations called embeddings.¹ When needed, the agent searches that information, receives the chunks deemed relevant to its task and uses them to generate a response. Typically, it can perform a lexical search (looking for specific terms), a semantic search (looking for chunks whose meanings are similar to what the task asks for), or a hybrid search (both). This is excellent when you need to find specific information in very large documents, as in the classic example of extracting relevant information from a 500-page contract. However, even if the agent formulates the search well, it may not receive all the chunks needed to address the task, leaving it with incomplete context. Even so, this technique remains widely discussed in both industry and academia, and it has been used in practically every implementation I have seen.

But it is not the only way to do it. One of the most frequently referenced names in the world of generative AI, Andrej Karpathy, proposed LLM Wiki.² This approach uses Markdown files (.md), which are simply text files with a particular formatting syntax, to maintain an index and interlinked pages of knowledge synthesised from sources. The agent can consult the relevant pages in full whenever it needs them. It is the equivalent of opening the user manual for a new object and, choosing from the index, reading only the references that matter to me. This has been my preferred approach for several use cases, because of its consistency and how easy it is to maintain or adjust manually. In complex use cases, though, it can create a greater maintenance burden than, for example, generating embeddings for a new version of a document and updating the index. We can build information-distillation pipelines (by which I mean reducing the amount of text while preserving information density) to maintain an LLM Wiki. That is, in fact, what Karpathy proposes. In my experience, however, models are still quite fallible at this distillation, frequently missing nuances in the original information and the overall purpose of the document. And the curation effort quickly becomes onerous.

Finally, since this piece is getting long and I do not want to discourage anyone who has read this far, I would like to briefly cover GraphRAG, or Graph Retrieval-Augmented Generation. Perhaps we should start with what a graph is: a mathematical or computational structure made up of a set of vertices (nodes) connected by edges (or links)... pause for breath. Put simply, it is a way of representing information and the relationships between data, as shown below. A good example is a social network, where the names of several people may be connected, but the links have a direction: Joaquina follows Manuel, who follows Ramiro, who in turn follows Joaquina. That does not mean Manuel follows Joaquina, or that Joaquina follows Ramiro. The people are nodes; the links are the follows. Information can have direction, and the relationships are explicit.

An arrow from A to B means “A follows B”. The reverse relationship is not implied.

GraphRAG uses a knowledge graph, with entities (concepts, objects, people) and relationships between them, to create a web that can be queried. It makes it easier for an agent to follow a research path through those relationships, looking for information to complete a task. The information found is then passed to the model as context to produce the response. This capability can be particularly useful when you need to combine information from several documents. As with the other approaches, the information and relationships need curation, especially when they are produced autonomously by the agent. For example, if the relationships are wrong, incomplete or out of date, the agent may leave out important information or base its answer on connections that no longer make sense. I have used this approach and ended up with a network so complex that neither I nor the agent (shoutout to Sr. Alfredo) understood what was in it.

These three approaches are not mutually exclusive, and a relatively complex enterprise implementation with a mature approach could, at the limit, use all three, depending on the case. A wiki can be searched with RAG, and GraphRAG itself is a way of supporting generation with retrieved information. Here is a summary of my mental model for when to use each one:

But context management does not end here. This is just the first article in a series in which I will explore the subject. In future pieces, we will discuss caching, compaction, context pruning and context isolation with subagents. See you soon.