Why Agents Need a Knowledge Base: The Gaps an LLM Can't Fill by Itself

When a team starts building an agent, the first version is very often only a model, a system prompt and a few tools, and for a demo that is usually enough, because the questions in a demo are general questions and a large language model knows a surprising amount about general things. The problems start when the agent is put in front of real work and the questions become specific: which services depend on this database, what we decided about retries last spring, whether the version of a library we use already has the fix, or how to apply a rule from a specialised engineering field that the model only knows from the textbook level. In my view, when an agent disappoints, the reason is very often that the model is asked to know things it never had any chance to learn, much more often than any weakness of the model itself. In this article I want to explain what those gaps are, why a bigger context window doesn't close them, and what role a knowledge base plays in an agentic system, so that the more technical articles on this site about RAG, agent memory and knowledge graphs have a place to start from.

What the model actually knows

Everything a model knows on its own is stored in its weights, and it got there during training. The authors of the original RAG paper, Patrick Lewis and his colleagues at Facebook AI Research, called this parametric memory, and already in 2020 they pointed out the two problems with it which are still the important ones today: for these models, "providing provenance for their decisions and updating their world knowledge remain open research problems" [1]. Their answer was to combine the parametric memory of the model with a non-parametric memory, which in their case was a dense vector index of Wikipedia that a retriever searched before the model generated an answer.

That distinction is very useful, because it gives us a clean way to think about what belongs where. Parametric memory is very good at language, reasoning, code patterns and widely published general knowledge. Non-parametric memory, which is the knowledge base, is for everything that is specific, private, recent or has to be traceable to a source.

The first gap: time

Every model has a date after which it knows nothing, and model providers publish that date. Anthropic's model overview, for example, lists two dates for each model: a reliable knowledge cutoff, which it describes as "the date through which the model's knowledge is most extensive and reliable", and a training data cutoff, which is the broader range of data that was used [2]. For the whole current Claude lineup, from Haiku 5.5 to Fable 5.1, both dates are June 2026 [2].

That sounds recent, and it is, but it also means that a model you call today in October 2026 has never seen anything from the last four months: not the NuGet packages released since then, not the breaking changes in the SDK you upgraded last week, and not the security advisory that was published yesterday. Additionally, the knowledge close to the cutoff is the thinnest, because the internet had not yet written much about those events when the data was collected. So the newest things are exactly the things the model knows least well, and these are very often the things a developer is asking about.

The second gap: knowledge that is too specialised

The gap that surprises people most is not about time at all. Many fields are old and stable, so the cutoff doesn't matter for them, and still the model's knowledge of them is very thin, simply because not much has been written about them in public. A model learns a fact roughly in proportion to how often that fact appears in its training data, and this has been measured. Nikhil Kandpal and his colleagues found that "a language model's ability to answer a fact-based question relates to how many documents associated with that question were seen during pre-training", and that models "must be scaled by many orders of magnitude" to answer well the questions with little support in that data [3]. Alex Mallen and his co-authors reached the same conclusion from a different direction: language models "struggle with less popular factual knowledge", and "scaling fails to appreciably improve memorization of factual knowledge in the long tail" [4].

That means there is a difference between general knowledge of a field and enough knowledge to do real work in it. A model has read the textbook level of almost everything, so it can explain the concepts of a specialised field very well, and this is exactly what makes the gap difficult to notice. When the question goes one level deeper, to the details that are needed to actually design or build something, the model answers in the same fluent and confident tone, but the answer is now built from general patterns and not from knowledge of that field. Some examples of areas where this happens:

• Designing a production line. The concepts, takt time, line balancing, bottlenecks and buffers, are in every operations management book, and a model explains them very well. But turning them into a line that works for a specific product needs the cycle times and changeover behaviour of specific machines, the constraints of a specific factory floor and the judgment of engineers who have commissioned such lines before, and most of that lives in vendor datasheets, simulation models and people's heads, not on the public internet.
• 3D modelling and design for manufacturing. A model can explain what a mesh, a B-rep solid or a tolerance stack is, but the rules that decide whether a part can actually be produced, such as the minimum wall thickness or the draft angle for one process and one material, or the scripting API of one particular CAD system, are mostly found in paid courses, vendor documentation and supplier guidelines, and there are very few public forum answers for the model to learn from.
• Industrial automation. PLC programs written in the IEC 61131-3 languages, like Structured Text and ladder logic, run a very large part of the world's factories, but almost all of that code is proprietary and stays inside the plant, so the amount of public example code is tiny compared with C# or Python, and each vendor adds its own dialect and function block libraries on top.
• Standards and regulation. Most ISO and IEC standards are sold as paid documents, so a model has usually seen blog posts and summaries about a standard but not the text of the standard itself. Local regulation has a similar problem when it is published only in a smaller language, for example national building or tax rules in Swedish, where the amount of text is a small fraction of what exists in English.
• Legacy and small software ecosystems. The customisation language of an old ERP system, the job control of a mainframe, or an open source library with a few hundred users all have very little public code. This is where a model very often invents method names and configuration keys that look completely plausible but don't exist.
• Tacit knowledge. How one particular machine behaves when the humidity changes, or what an experienced operator listens for before a failure, was never written down anywhere, so no model can learn it, however large it becomes.

Some of these gaps will become smaller as models grow and as more is written in public, but in my view many of them will not close, because the knowledge is not missing from the training data by accident: it is paid, proprietary, in a small language, or never written down at all, and the research above shows that scaling the model alone is a very expensive way to fix the long tail. The same research also points to the solution. Kandpal and his colleagues note that retrieval "can reduce the dependence on relevant pre-training information" [3], and Mallen and his co-authors found that retrieval-augmented models "largely outperform orders of magnitude larger LMs" on less popular knowledge [4]. In other words, for a specialised field the knowledge base is where the depth comes from: the vendor manuals, the licensed standards, the internal design rules, the commissioning reports and the notes of experts, while the model contributes the reasoning and the language.

The third gap: knowledge that was never public

The specialised fields above are at least partly public, so a model has seen something about them, but a model is trained only on public data, and most of what an agent needs to know inside a company has never been public and never will be. The model doesn't know that your billing-api writes to the same PostgreSQL cluster as invoicing-worker, it doesn't know which team owns the notifications service, it doesn't know the architecture decision record in which you chose at-least-once delivery and explained why, and it doesn't know about an incident last March whose root cause was a connection pool that was too small.

No future model will close this gap by being trained on more data, because the data is not available to anyone who trains models, and that's exactly how it should be. The only way the agent gets this knowledge is if we give it to the agent at the moment it needs it.

The fourth gap: knowing where an answer came from

The fourth gap is the one from the RAG paper: provenance [1]. When a model answers from its weights, it can't tell you which document the answer came from, because there was no document, there were only weights. For a casual question that is fine, but for an agent that is going to change a configuration, open a pull request or tell an on-call engineer which service to restart, an answer without a source is very difficult to trust and impossible to check.

A knowledge base changes this, because every piece of knowledge that is retrieved comes with a source, a date and an owner. The agent can say "according to ADR-0042, updated on 3 September" and a human can open ADR-0042 and see if the agent understood it correctly. In my view this is the most underrated benefit of a knowledge base, because besides giving the agent more knowledge it also makes that knowledge checkable by the people who have to act on it.

Why a bigger context window doesn't solve it

The natural question is whether we still need any of this, now that context windows are so large. All current Claude models have a 1M token context window, which Anthropic estimates at roughly 555k words on the current tokenizer [2]. Why not put all the documentation into the prompt and let the model find what it needs?

There are three reasons why this doesn't work as well as it sounds. The first one is that long context is not read evenly. In the paper Lost in the Middle, Nelson Liu and his co-authors showed that "performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts" [5]. The paper is from 2023 and models have changed since then, but the lesson is still valid: putting more text in front of the model doesn't guarantee that the model actually uses all of it.

The second reason is cost and latency, because every token in the context is paid for on every call, and an agent makes many calls in a single task. Your AI Cost Model Is Already Wrong shows how quickly this adds up, and a context that is mostly irrelevant documentation is the most expensive way to make an agent slower.

The third reason is that even a 1M token window has to be filled by something, and that something has to decide what is relevant, what is current and what is allowed to be seen by this user. That something is a knowledge base with a retrieval step, whether we call it that or not.

The three shapes a knowledge base takes

"Knowledge base" can mean very different things, and in my view a lot of confusion comes from treating them as one. In an agentic system it's very useful to separate three shapes, because each one answers a different kind of question.

The document store is the one most teams build first, and it's very good for questions whose answer is written down somewhere in one place. It is weak at questions whose answer is spread over many places and only appears when you follow relationships, which is where the knowledge graph comes in. A vector index can tell the agent which documents look similar to "billing database", but it can't tell it that four services depend on that database through two hops, because similarity is not dependency.

The knowledge graph also helps with a kind of question that plain RAG handles badly, which is the global question about a whole collection. Darren Edge and his colleagues at Microsoft Research described this in the GraphRAG paper: standard RAG struggles with "global questions directed at an entire text corpus, such as 'What are the main themes in the dataset?'", and their approach first uses an LLM to "derive an entity knowledge graph from the source documents" and then pre-generates summaries for groups of closely related entities [6]. You don't need GraphRAG itself to benefit from the idea, but it shows very clearly that the structure of a knowledge base decides which questions it can answer.

Agent memory is the third shape and the most different one, because here the agent is not only a reader but also a writer. It records what it learned and what was decided so that the next session doesn't start from zero. That makes it very powerful, but it also means it can fill itself with stale or contradicting facts, which is the subject of Keeping Memory True.

In practice a serious agent ends up using more than one of these, and the retrieval step combines them. Hybrid Retrieval in C# shows that pipeline in full: vector search finds the entry points, graph traversal expands the context around them, and a reranker decides what actually reaches the model.

Let the agent decide when to look

There are two ways to connect a knowledge base to a model. In classic RAG, retrieval runs before every model call, whether the question needs it or not. In an agentic system it's usually better to expose the knowledge base as a tool, so that the model decides when it needs to look something up, what to search for and whether the first result was enough or it should search again with different words.

In .NET, Microsoft.Extensions.AI makes this very convenient. AIFunctionFactory.Create turns a normal .NET method into a tool the model can call, and if you don't pass a description it takes it from the [Description] attribute on the method [7]. UseFunctionInvocation() then adds a FunctionInvokingChatClient to the pipeline, which runs the tool when the model asks for it and sends the result back [8].

using System.ComponentModel;
using Microsoft.Extensions.AI;
 
public sealed record KnowledgeHit(
    string Text,
    string Source,          // e.g. "adr/0042-at-least-once-delivery.md"
    DateTimeOffset UpdatedAt,
    double Score);
 
public sealed class KnowledgeTools(IKnowledgeStore store)
{
    [Description("Searches the team's knowledge base: architecture decisions, service " +
                 "ownership and dependencies, runbooks and incident reports. Use it before " +
                 "answering anything about our own systems. Each hit has a source and a date.")]
    public Task<IReadOnlyList<KnowledgeHit>> SearchKnowledge(string question, int top = 5) =>
        store.SearchAsync(question, Math.Clamp(top, 1, 8));
}

The important design decision is in the return type. Every hit carries its source and the date it was last updated, because without them the agent can't give us the provenance that was the whole point, and it can't tell an old runbook from a current one.

var knowledge = new KnowledgeTools(store);
 
IChatClient agent = chatClient
    .AsBuilder()
    .UseFunctionInvocation()
    .Build();
 
var options = new ChatOptions
{
    Tools = [AIFunctionFactory.Create(knowledge.SearchKnowledge, name: "search_knowledge")]
};
 
List<ChatMessage> messages =
[
    new(ChatRole.System, """
        You answer questions about our services. Facts about our own systems must come
        from search_knowledge, and you must name the source and its date for each one.
        If the knowledge base has nothing on the question, say so instead of guessing.
        """),
    new(ChatRole.User, "What breaks if we take billing-db offline on Sunday night?")
];
 
ChatResponse response = await agent.GetResponseAsync(messages, options);
Console.WriteLine(response.Text);

The last sentence of the system prompt matters very much. A model that is told to use the knowledge base but is not told what to do when it finds nothing will very often fall back to its parametric memory and produce a plausible answer about a system it has never seen, and that is exactly the failure the knowledge base exists to prevent.

If more than one agent or tool needs the same knowledge, the next step is to put the search behind an MCP server instead of a method in one application, so that Claude Code, an IDE assistant and your own agents all use the same source. Writing an MCP Server in C# covers that part.

The risk of a stale knowledge base

There is one risk that is easy to miss. When an agent answers from its weights and is wrong, people have learned to be suspicious. When it answers from the knowledge base and cites a source, people trust it, so if the source is out of date the agent becomes confidently and convincingly wrong. A service dependency that was removed six months ago but is still an edge in the graph will send an engineer to the wrong place at three in the morning, and the citation will make it look correct.

This is why maintenance has to be part of the design from the first version. Every fact should have a source and an updated date, facts that are derived from systems (deployment manifests, git history, tracing data) should be re-derived on a schedule rather than written by hand, and conflicts between sources should be visible instead of silently resolved. Keeping Your Knowledge Graph Current Without a Dedicated Team builds exactly this loop for a knowledge graph.

Where to start

If you are building an agent and you are not sure if you need a knowledge base, a simple test is to write down ten real questions the agent should answer and mark for each one whether the answer is general knowledge, recent, specialised, private, or needs a source. If most of them are general, the model alone is probably enough. If most are specialised, private or need a source, which in my view is the case for almost every agent that does real work inside a company, then the knowledge base becomes the main part of the system, and the model becomes the part that reads it and reasons about it.

From there the order that makes sense is usually:

1. Start with a document store and vector search over the documents you already have, and return the source and date with every chunk.
2. Expose it as a tool and tell the agent explicitly what to do when the search finds nothing.
3. When the questions start to be about relationships ("what depends on", "who owns", "what changed together"), add a knowledge graph next to it rather than trying to make embeddings answer them.
4. Add agent memory when the agent needs to carry decisions from one session to the next, and give it decay and verification from the beginning.
5. Automate the maintenance before you have many users, because a stale knowledge base loses trust very quickly and it is very difficult to win it back.

The model gives an agent its ability to reason and to write, and that part keeps getting better with every release, but knowledge of your own systems, your decisions and the last few months has to come from somewhere else, and in an agentic system that place is the knowledge base, together with the work of keeping it current.

References

1. Lewis, P. et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020. https://arxiv.org/abs/2005.11401
2. Anthropic. Models overview, Claude Platform documentation, accessed 16 October 2026. https://platform.claude.com/docs/en/about-claude/models/overview
3. Kandpal, N. et al. Large Language Models Struggle to Learn Long-Tail Knowledge, ICML 2023. https://arxiv.org/abs/2211.08411
4. Mallen, A. et al. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories, ACL 2023. https://arxiv.org/abs/2212.10511
5. Liu, N. F. et al. Lost in the Middle: How Language Models Use Long Contexts, 2023. https://arxiv.org/abs/2307.03172
6. Edge, D. et al. From Local to Global: A Graph RAG Approach to Query-Focused Summarization, 2024. https://arxiv.org/abs/2404.16130
7. Microsoft. AIFunctionFactory.Create Method, .NET API reference. https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.ai.aifunctionfactory.create
8. Microsoft. Use the IChatClient interface, .NET documentation. https://learn.microsoft.com/en-us/dotnet/ai/ichatclient