Building a Retrieval-Augmented Generation (RAG) System in .NET

Introduction

Retrieval-Augmented Generation (RAG) is the pattern most teams reach for when they want an LLM to answer from their own data. Most RAG code lives in Python, but the .NET AI stack has caught up enough that you can build the whole pipeline, embedding, retrieval and generation, inside .NET.

In this article I walk through how we designed and built a local RAG architecture in C#. It started as a proof of concept and later became a production Background Verification System.

We wanted something fully local and easy to change, built on:

• LLamaSharp for running the language model on our own hardware
• ONNX Runtime for generating embeddings
• PgVector for similarity search inside PostgreSQL

Along the way you will see:

• How to design a local RAG pipeline in .NET
• Which libraries and NuGet packages give you tokenization, vector search and GPU acceleration
• How we used PostgreSQL and PgVector for similarity search
• What we learned about performance, scale and maintenance in a .NET AI system

By the end you should have a clear picture of how the pieces fit together, and how the design changes when you move from an experiment to something you run in production.

Components of the RAG Implementation

The system has four main parts, each responsible for one stage:

1. Embedding Service. Turns documents and queries into vectors.
2. Document and Knowledge Store. Holds structured data such as user profiles, company records and insurance information in PostgreSQL with vector support.
3. LLM Engine. Runs local inference and writes the answer from the retrieved context.
4. API Layer and Retriever. Takes the request and drives the whole flow, with endpoints for chat, search and lookups.

The Embedding Service

The first thing we built was the Embedding Service. This is where semantic search starts. It takes raw text, either a document or a user query, and turns it into a vector that carries the meaning of that text.

Before you can generate an embedding you have to tokenize the text, which means breaking it into small pieces the model understands, such as words or parts of words. The tokens then go into an embedding model, for example a BERT model exported to ONNX, and it returns a high dimensional numeric vector.

NuGet Packages Used

• Microsoft.ML.Tokenizers A native BERT style tokenizer for .NET. It does modern NLP tokenization inside your .NET application with no Python dependency, and it matches the vocabulary of common transformer models.
• Microsoft.ML.OnnxRuntime An inference engine for ONNX models in .NET. It runs embedding generation and other model work on CPU or GPU.

Together these two give you embedding generation that runs entirely inside the .NET runtime.

Embedding Terminology

Tokenize

Splitting text into smaller parts called tokens, which can be words, parts of words, or single characters. Tokenization prepares the text so the model can process it. Transformer models such as BERT need a tokenizer that matches their own vocabulary.

Embedding

An embedding is a numeric vector that represents the meaning of a piece of text. It lets you measure how close two pieces of text are, run semantic search, and feed other AI systems. It is the bridge between text and meaning.

BERT

BERT (Bidirectional Encoder Representations from Transformers) is a transformer model from Google. It reads text in both directions, left to right and right to left, so it picks up context more accurately. BERT and similar models are often exported to ONNX so they can run efficiently in .NET through Microsoft.ML.OnnxRuntime.

ONNX

Open Neural Network Exchange (ONNX) is an open standard for moving machine learning models between frameworks. A model trained in PyTorch or TensorFlow can be exported and then run somewhere else, including inside a .NET application with Microsoft.ML.OnnxRuntime. You get to use transformer models from C# without Python and without the original training environment.

Inside the Embedding Service

Once the tokenizer and the ONNX model were configured, embedding generation happens in four steps.

Input Validation and Tokenization

The service checks that the input text is valid, then converts it into token IDs with the BERT tokenizer.

Sequence Preparation: Padding, Truncation and Attention Mask

Transformers need fixed length input. We pad or truncate the tokens to the maximum sequence length, and build an attention mask so the model knows which tokens are real and which are padding.

Tensor Creation and Model Inference

We turn the arrays into tensors, feed them to the ONNX model, and run inference in the ONNX Runtime session.

Extracting the Embedding Vector

After inference we take the output tensor that holds the embedding and convert it to a float array.

Document and Knowledge Store

The difference between a RAG system and a plain LLM is where the knowledge comes from. A plain LLM answers only from what is inside its own parameters. A RAG system first retrieves external context, meaning facts, documents or structured data from a knowledge base, and then generates the answer.

That gives you answers that are grounded and current, without retraining or fine tuning anything.

For this we needed a database that can store and search vector embeddings, the numeric representation of documents and queries that the Embedding Service produces. Each vector carries the meaning of a piece of text, so you can find related documents even when the words do not match.

We used PostgreSQL with PgVector, an extension that adds vector similarity search to Postgres. That gave us the reliability of a relational database and the semantic search we needed in one place.

Why PostgreSQL with PgVector?

• Familiar to .NET developers. PostgreSQL works well with Entity Framework Core.
• Native vector operations. PgVector supports cosine similarity, Euclidean distance and inner product directly in SQL.
• One storage system. No separate vector database. Embeddings, metadata and documents all live together.
• Scale. PostgreSQL handles large datasets and brings mature indexing and transactions with it.

NuGet Packages Used by the Knowledge Store

• Pgvector Adds the vector data type and the similarity operators to PostgreSQL so you can store, index and query embeddings in the database. This is what gives you cosine similarity and Euclidean distance in SQL.
• Pgvector.EntityFrameworkCore Connects pgvector to Entity Framework Core so embeddings behave like normal entity fields in C#. You can run similarity searches and persist records with the EF Core patterns you already know, without dropping to raw SQL.

Vector Terminology

Understanding a few vector concepts makes the rest of the system much easier to follow.

Vector Data Type

A vector data type lets you store a list of floating point numbers in a single database column. Those numbers are usually the embedding produced by a model.

An embedding for a piece of text might be 384 or 768 float values, all in one row and one column. PostgreSQL with pgvector can store, index and query those values efficiently.

Vector Search

Vector search, also called similarity search, finds the records whose embeddings are closest to a query vector.

Instead of matching keywords, it measures how close two pieces of text are in meaning, using a distance function such as:

• Cosine similarity
• Euclidean distance
• Inner product

That is what lets you find the right document, profile or knowledge base entry even when the query does not use the same words as the stored text.

In short: keyword search finds documents that look similar. Vector search finds documents that mean something similar, and that is the part RAG depends on.

Inside the Postgres Gateway

Vector-Enabled Database with Entity Framework Core

We used Entity Framework Core as the data layer for PostgreSQL. The Pgvector.EntityFrameworkCore extension made this easy, because it enables the vector type directly in EF models.

To turn vector support on, chain .UseVector() onto UseNpgsql().

That tells EF Core to recognise the PostgreSQL vector type, so you can store and query embeddings as part of your entities.

Populating the Database with Embeddings

When we processed person data, we first gathered the context that belongs to that person, such as company affiliations, travel history and insurance details, into one piece of text. Then we generated the embedding for it.

Here is a simplified version of how we built and stored the embedding for each record.

LLM Engine

The LLM Engine writes the final answer. Once the Knowledge Store has returned the relevant documents, the engine takes the original query plus the retrieved context and produces the response.

This is the reasoning layer. The embedding and retrieval stages decide what information comes in. The LLM Engine decides how it is said.

We ran the model locally to keep the data private and to avoid depending on a cloud API. It runs inside the .NET application with LLamaSharp and ONNX Runtime, on local hardware.

NuGet Packages Used by the LLM Engine

• LLamaSharp A .NET library for running Llama family models natively in .NET. It gives you APIs for:

  • Loading and initialising models
  • Running inference
  • Talking to a local LLM

• LLamaSharp.Backend.Cuda12.Windows An optional GPU backend for LLamaSharp that uses CUDA 12 on Windows. It puts an NVIDIA GPU to work, which matters a lot with large models and anything close to real time.

Is LLamaSharp Developed by Meta?

No. LLamaSharp is an independent open source project.

It is a .NET library built and maintained by independent contributors on GitHub, not by Meta.

The goal is to let C# and .NET developers load and run Llama family models, and other compatible models, inside .NET applications with no Python and no external runtime.

Under the hood it uses llama.cpp, the community C and C++ implementation of the Llama inference engine. That is why it supports any model that works with the llama.cpp backend format.

Supported Models

LLamaSharp works with:

• Llama 1, 2, 3 and 4

and also with many other transformer models converted to GGUF or GGML, including:

• Mistral
• Phi
• Vicuna
• Alpaca

In short: LLamaSharp is not made by Meta. It is a .NET door into the open Llama ecosystem, so you can run modern LLMs locally.

About Model Formats: GGML and GGUF

When Meta releases the Llama models they come as PyTorch weight files, the raw checkpoints. To run them outside Python, the community converts those weights into lighter, portable formats such as GGML and GGUF.

• GGML and GGUF are binary model formats built for fast inference on CPU and GPU.
• GGUF is the newer one. It is more flexible, better structured, and handles model metadata and quantization better.
• Both are supported by llama.cpp, and therefore by LLamaSharp, which is what makes local inference in C# possible.

In practice, a .gguf file you download is already prepared for CPU or GPU use with LLamaSharp or Ollama.

Understanding Tokens in LLMs

What is a Token?

A token is a unit of text the model works with. It can be:

• A whole word
• Part of a word
• A single character

Example: "WeAreDeveloper!" becomes [ "We", "Are", "Dev", "eloper", "!" ], which is 5 tokens.

Why Tokens Matter

• Billing. With a cloud model such as OpenAI or Azure OpenAI you pay per token, for both the prompt and the response. More tokens, higher bill.

• Model limits. Every model has a maximum token count per request, for example 512, 4,096, or 128,000 and more. If your prompt plus the response goes over that limit, the model truncates or rejects the input.

Even with local inference through LLamaSharp, tokens still matter. They decide how much context the model can see, which shows up directly in the quality of the answer.

Token Performance and Limits

The number of tokens in a prompt affects speed, memory and the quality of the answer. Getting that balance right is most of the work in a RAG pipeline.

Shorter Prompts, Fewer Tokens

• Faster inference
• Less memory and compute
• Good for short, direct questions
• Less context to work with

Longer Prompts, More Tokens

• Richer context and longer conversations
• Summarising or answering over large documents
• Slower, and heavier on resources

What Different Token Limits Mean

• Small limits, 128 to 256 tokens. The model keeps less context and may lose the earlier part of a conversation. Fast and cheap, but not usable for large documents or multi turn reasoning.
• Large limits, 4,096 tokens and above. The model can work over a much larger context window, which you need for summarising or reasoning over long text. Slower and heavier, especially on a local GPU.

Tip. Balance context size against inference speed. For most local setups, 1,024 to 2,048 tokens is a good middle ground.

LLM Terminology

Model

Model files, for Llama, GPT or anything else, hold the learned parameters of the network after training:

• Weights. Numbers that describe every connection between layers. They decide how input tokens turn into output.
• Biases. Extra parameters in each neuron or layer that shift the activations and help the model learn.

Prompt

The prompt is the input or instruction you give the model. It can be a question, a statement, or a structured set of instructions.

Temperature

Temperature controls how random the output is.

• Low, for example 0.1. Deterministic, focused, consistent.
• High, for example 1.0. More varied, more creative, less predictable.

Instructive Models

An instructive model is an LLM fine tuned to follow instructions. Models such as GPT-3.5-turbo-instruct or Llama-2-Chat are trained to understand and carry out what you ask, so the output lines up better with the request.

Inside the LLM Engine Service

Method Signature

The main inference method is asynchronous and returns a stream of generated text. Its parameters control how the model behaves.

Parameters:

• prompt. The user input or instruction.
• antiPrompts. Phrases that stop generation early if the model produces them.
• maxTokens. The maximum number of tokens to generate.
• temperature. How random the output is.

Sampling Pipeline Configuration

The sampling pipeline decides how the model picks the next token at each step.

Settings:

• Temperature. Randomness in generation.
• TopK and TopP. How wide the sampling is, either from the top K most likely tokens or from a probability mass P.
• Seed. Makes the output reproducible.

Inference Parameters Setup

All the configuration values go into a single object, which keeps the call site clean and puts every inference setting in one place.

Streaming Inference

The model generates text piece by piece and returns it as an asynchronous stream. Each chunk goes to the caller straight away, so a UI or a console can update live.

This is the same experience you get from the OpenAI chat APIs, and it makes a noticeable difference in anything user facing.

API Layer and RAG

Once the three core parts are in place, the Knowledge Store, the LLM Engine and the Embedding Service, putting Retrieval-Augmented Generation together is mostly wiring.

What Is RAG?

Retrieval-Augmented Generation combines two things:

Retrieval

The system searches an external database or knowledge store for the information that is most relevant to the query, usually with vector search against precomputed embeddings.

Generation

That information goes to a Large Language Model together with the user query, and the model writes the final answer.

Why RAG?

A plain LLM answers only from its training data, so it cannot see anything fresh, domain specific or private. RAG fixes that by letting the model look up the data it needs, which makes the answer more accurate and easier to trust.

Inside the RAG Service

Here is the flow in the .NET service:

1. Receive query. The API takes the user query.
2. Generate embedding. The query becomes a vector.
3. Vector search. The system searches PostgreSQL with pgvector for the closest records.
4. Context assembly. The matching records are fetched and turned into context.
5. LLM generation. Context and query go to the LLM Engine, which streams back the answer.

The API sits between retrieval and generation, and that is what makes the answers grounded in real data instead of guesses.

Finding the Most Relevant Person

Retrieval is the heart of it: finding the right records for the query. Here the system runs a hybrid search, vector first and then LLM based ranking, to find the person record that fits best.

1. Embedding generation. The query becomes a vector through the Embedding Service.
2. Vector search. That vector goes into a similarity search in PostgreSQL with pgvector to get the top candidates.
3. Result ranking. Candidates are ordered by cosine similarity.
4. Context preparation. The best results go to the LLM together with their context for a closer look.

Only the records that are genuinely close in meaning make it to the final step.

Generating the Final Answer with the LLM

With the context in hand, the LLM writes the answer.

1. Prompt building. The system builds a prompt from the query and the retrieved context.
2. Streaming inference. The LLM Engine is called as an asynchronous stream and sends partial results as they arrive.
3. Live output. That lets the UI or the API update in real time.

Retrieval and generation together close the Retrieval-Augmented Generation loop inside the .NET application.

Bringing It All Together

With a vector database, local LLM inference and a retrieval layer on top, you get a complete RAG system that runs entirely in .NET.

The pipeline is simple:

1. Data storage and embeddings. People, companies and documents live in PostgreSQL with pgvector, which gives you semantic search over them.
2. LLM engine. LLamaSharp runs the model locally with no Python, and a CUDA backend keeps inference fast enough to feel interactive.
3. RAG layer. The API generates the query embedding, retrieves what matters, and passes it to the model to write the answer.

What this shows is that you can build this kind of system natively in C#, with open source tooling, local inference and vector search, and keep the whole thing self contained.

The result turns a static database into something you can ask questions of: it understands the query, finds the right context, and answers in plain language, all on the .NET platform.

If you want to look at the implementation, the full source is on GitHub: dotnet-talk.ai.profile.finder

The repository has every part of the pipeline, embedding generation, vector search with PostgreSQL and PgVector, and local LLM inference with LLamaSharp, plus examples you can run and build on.