AI/September 23, 2026/7 min read

Building an Agentic System in .NET, Part 3 — Durable Memory with Postgres and pgvector

Give your .NET agent a long-term memory it can actually recall: model MemoryItem in EF Core, generate embeddings with Microsoft.Extensions.AI, store them in Postgres with pgvector, and build an HNSW cosine index with a ranked recall query that blends similarity, recency, and importance.

.NET Agent Framework AI EF Core pgvector Postgres

The previous two parts of this series wired up the agent loop and gave it tools. The missing piece is memory. Session context resets with every new conversation; anything the agent learned about a user, their preferences, or a prior decision vanishes. Long-term memory fixes that — but only if you store the right things, index them properly, and inject them at the right moment.

This part covers all of that with working EF Core code, a real HNSW index, and a recall query you can drop into a production system.

What Belongs in Long-Term Memory

The most common mistake is storing raw conversation turns. Turn-by-turn transcripts grow without bound and are full of noise — small talk, clarifications, reformulations. They belong in a separate, time-bounded session store (Redis with a TTL works fine).

Long-term memory should hold distilled facts: things that are both durable and worth retrieving across sessions. Good candidates:

User preferences expressed as explicit choices ("always use metric units")
Decisions the agent helped make ("migrated to Dapr in sprint 14")
Domain facts the user provided that aren't in the base model
Corrective feedback ("I said the deadline is Friday, not Thursday")

Each item gets an importance score at write time (0.0–1.0) and a timestamp. Both feed into recall ranking later.

The EF Core Entity and Migration

Add the packages first:

dotnet add package Pgvector.EntityFrameworkCore --version 0.3.0
dotnet add package Npgsql.EntityFrameworkCore.PostgreSQL --version 10.0.3

Note the version constraint: Pgvector.EntityFrameworkCore v0.3.x targets EF Core 9 and 10. If you're still on EF Core 8, pin to v0.2.2.

using Microsoft.EntityFrameworkCore;
using Pgvector;
 
public class MemoryItem
{
    public Guid Id { get; set; } = Guid.NewGuid();
    public string UserId { get; set; } = string.Empty;
    public string Content { get; set; } = string.Empty;
    public Vector Embedding { get; set; } = null!;
    public float Importance { get; set; }          // 0.0 – 1.0
    public DateTimeOffset CreatedAt { get; set; } = DateTimeOffset.UtcNow;
    public DateTimeOffset LastAccessedAt { get; set; } = DateTimeOffset.UtcNow;
}
 
public class AgentDbContext(DbContextOptions<AgentDbContext> options)
    : DbContext(options)
{
    public DbSet<MemoryItem> Memories => Set<MemoryItem>();
 
    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.HasPostgresExtension("vector");
 
        modelBuilder.Entity<MemoryItem>(e =>
        {
            e.HasKey(m => m.Id);
            e.Property(m => m.Embedding).HasColumnType("vector(1536)");
            e.HasIndex(m => m.UserId);
        });
    }
}

builder.Services.AddNpgsql<AgentDbContext>(
    connectionString,
    npgsqlOptions => npgsqlOptions.UseVector());

Generate the migration with dotnet ef migrations add AddMemoryItem. The generated migration will create the table with a vector(1536) column, but EF Core has no first-class HNSW index fluent API. You must append the index DDL manually inside the migration:

public partial class AddMemoryItem : Migration
{
    protected override void Up(MigrationBuilder migrationBuilder)
    {
        migrationBuilder.Sql("CREATE EXTENSION IF NOT EXISTS vector;");
 
        migrationBuilder.CreateTable(
            name: "Memories",
            columns: table => new
            {
                Id            = table.Column<Guid>(nullable: false),
                UserId        = table.Column<string>(nullable: false),
                Content       = table.Column<string>(nullable: false),
                Embedding     = table.Column<Vector>(type: "vector(1536)", nullable: false),
                Importance    = table.Column<float>(nullable: false),
                CreatedAt     = table.Column<DateTimeOffset>(nullable: false),
                LastAccessedAt = table.Column<DateTimeOffset>(nullable: false)
            },
            constraints: t => t.PrimaryKey("PK_Memories", x => x.Id));
 
        // EF Core has no fluent API for HNSW — add it directly.
        migrationBuilder.Sql(
            """CREATE INDEX ON "Memories" USING hnsw ("Embedding" vector_cosine_ops)
               WITH (m = 16, ef_construction = 64);""");
 
        migrationBuilder.CreateIndex(
            name: "IX_Memories_UserId",
            table: "Memories",
            column: "UserId");
    }
 
    protected override void Down(MigrationBuilder migrationBuilder)
        => migrationBuilder.DropTable("Memories");
}

HNSW Parameter Guidance

m = 16 is the default maximum connections per layer; ef_construction = 64 is the build-time search width. Higher values improve recall at the cost of longer build time and more memory. For an agent memory store where items arrive incrementally (not in one bulk load), HNSW is the right choice over IVFFlat — IVFFlat requires an ANALYZE training step after bulk inserts and degrades if you skip it.

Critical gotcha: HNSW in pgvector has a hard 2,000-dimension ceiling. text-embedding-3-small produces 1,536 dimensions — fits cleanly. text-embedding-3-large produces 3,072 dimensions and will fail at index creation with column cannot have more than 2000 dimensions for hnsw index. Stick to text-embedding-3-small, or use the halfvec column type if you need the larger model.

The Embedding Service

Since Agent Framework 1.0 (GA April 3, 2026) and the current Semantic Kernel both converge on Microsoft.Extensions.AI, write against that abstraction:

using Microsoft.Extensions.AI;
 
public interface IMemoryEmbeddingService
{
    Task<ReadOnlyMemory<float>> EmbedAsync(
        string text, CancellationToken ct = default);
}
 
public sealed class MeaiEmbeddingService(
    IEmbeddingGenerator<string, Embedding<float>> generator)
    : IMemoryEmbeddingService
{
    public async Task<ReadOnlyMemory<float>> EmbedAsync(
        string text, CancellationToken ct = default)
    {
        var result = await generator.GenerateAsync(
            [text], cancellationToken: ct);
        return result[0].Vector;
    }
}

// Azure OpenAI
builder.Services.AddAzureOpenAIEmbeddingGenerator(
    deploymentName: "text-embedding-3-small",
    endpoint: new Uri(config["AzureOpenAI:Endpoint"]!),
    credential: new DefaultAzureCredential());
 
builder.Services.AddSingleton<IMemoryEmbeddingService, MeaiEmbeddingService>();

Swapping to Ollama or another local model is a one-line DI change. Nothing else in the pipeline cares.

Chunking Before Storage

A single long paragraph makes a poor memory item — the embedding averages the whole thing and similarity scores become mushy. Chunk first. SemanticChunker.NET (by Gregor Biswanger) works with Microsoft.Extensions.AI and the recommended threshold is Percentile 95%. Start there, then tune in 5-point steps (90 → 95 → 98) depending on whether you're getting too many tiny fragments or too few meaningful ones. Leave headroom in the token budget: your 8,192-token model context needs space for the system prompt, the retrieved memories, and the new user turn.

Sequential Scan vs. HNSW: Why It Matters

At a demo scale of a few hundred rows, a sequential scan is invisible. At 10,000 rows it starts to sting; at 100,000 rows it is unacceptable in a user-facing path.

HNSW navigates a multi-layered graph from coarse to fine approximations; query complexity grows logarithmically, not linearly. The DBI-services benchmark (March 2026, using 25,000 Wikipedia articles embedded with text-embedding-3-large) illustrates the pattern: sequential scan is the baseline and HNSW must be forced on small tables by disabling enable_seqscan to even measure it. On realistic production data the gap is not subtle.

At 50M vectors, pgvectorscale (DiskANN-based) reaches 471 QPS at 99% recall; that comparison bracket is well beyond where plain pgvector is the right tool. For fewer than ~10M vectors with mixed relational and vector query patterns, pgvector with HNSW is the right call.

One tuning point that catches teams off guard: hnsw.ef_search defaults to 40. That caps the candidate list the index returns at query time. Always tune it with SET LOCAL inside a transaction:

BEGIN;
SET LOCAL hnsw.ef_search = 100;
SELECT ... FROM "Memories" ORDER BY ... LIMIT 10;
COMMIT;

Never use a session-level SET in a connection-pooled environment — the setting persists through pooler reuse and silently affects unrelated queries on that connection.

The Recall Query

Pure cosine similarity isn't enough. A memory from three years ago that scores 0.97 similarity might be less useful than a slightly less similar memory from last week that the user flagged as critical. The recall score blends three signals:

public async Task<IReadOnlyList<MemoryItem>> RecallAsync(
    string userId,
    string queryText,
    int topK = 5,
    CancellationToken ct = default)
{
    var queryVector = await _embedding.EmbedAsync(queryText, ct);
    var pgVector    = new Vector(queryVector.ToArray());
 
    // Weights: tune these per use-case
    const float wSimilarity = 0.6f;
    const float wRecency    = 0.2f;
    const float wImportance = 0.2f;
 
    // Recency: exponential decay, half-life = 30 days
    // EF Core translates CosineDistance to <=> operator
    var now = DateTimeOffset.UtcNow;
 
    var results = await _db.Memories
        .Where(m => m.UserId == userId)
        .Select(m => new
        {
            Item       = m,
            Similarity = 1f - m.Embedding.CosineDistance(pgVector),
            Recency    = (float)Math.Exp(
                -0.693f * EF.Functions
                    .DateDiffDay(m.LastAccessedAt, now) / 30.0),
        })
        .Select(x => new
        {
            x.Item,
            Score = wSimilarity * x.Similarity
                  + wRecency    * x.Recency
                  + wImportance * x.Item.Importance
        })
        .OrderByDescending(x => x.Score)
        .Take(topK)
        .Select(x => x.Item)
        .ToListAsync(ct);
 
    // Update last-accessed timestamp in a fire-and-forget update
    var ids = results.Select(m => m.Id).ToList();
    await _db.Memories
        .Where(m => ids.Contains(m.Id))
        .ExecuteUpdateAsync(s =>
            s.SetProperty(m => m.LastAccessedAt, now), ct);
 
    return results;
}

EF Core translates CosineDistance to the <=> pgvector operator. The HNSW index picks it up automatically. The recency decay uses a 30-day half-life; halve or double it depending on how ephemeral your domain is.

The SessionStart Injection Hook

The recall query is useless if nobody calls it. Wire it into a SessionStart hook — a middleware or agent lifecycle event that fires before the first user message reaches the model:

public sealed class MemoryInjectionMiddleware(
    IMemoryRecallService recall,
    ILogger<MemoryInjectionMiddleware> logger)
{
    public async Task<AgentContext> OnSessionStartAsync(
        AgentContext context, CancellationToken ct = default)
    {
        var memories = await recall.RecallAsync(
            context.UserId,
            context.InitialMessage,
            topK: 5, ct);
 
        if (memories.Count > 0)
        {
            var block = string.Join("\n",
                memories.Select((m, i) => $"[Memory {i + 1}] {m.Content}"));
 
            context.SystemPrompt = $"""
                {context.SystemPrompt}
 
                ## Recalled context from prior sessions
                {block}
                """;
 
            logger.LogInformation(
                "Injected {Count} memories for user {UserId}",
                memories.Count, context.UserId);
        }
 
        return context;
    }
}

The agent never sees raw database rows — it sees its own system prompt, enriched with the top-5 most relevant memories, ranked by the blended score. That's the complete feedback loop: distil facts at write time, rank by relevance plus recency plus importance at read time, inject before the model sees the first token.

What's Next

Part 4 takes on the other half of the problem: memory that is stored but no longer true. Timestamps and decay scoring, deduplication, contradiction detection, verification on read, and pruning as a hosted service, so the recall query you just built keeps returning facts the agent can still rely on.

Sources

September 19, 2026 · 6 min

Building an Agentic System in .NET, Part 2 — Writing an MCP Server in C#

Learn how to expose your own tools to any AI agent by building a Model Context Protocol server in C# using the official ModelContextProtocol SDK v2.2.0 — covering tool design, input validation, context window budgeting, and the naming mistakes that make models misuse your tools silently.

Read

September 18, 2026 · 7 min

Building an Agentic System in .NET, Part 1 — Anatomy of an Agent Harness

A chat wrapper is not an agent harness. This article walks through every structural component a real .NET 10 agent host needs — the turn loop, tool dispatch, persistence, and safety layer — and shows the minimal C# skeleton that survives a crash.

Read

July 16, 2026 · 6 min

Graph-Native Data Structures in C#, Part 4 — Graphs & Adjacency with a Social Follow Network

Part 4 of the series models a directed social follow network in NebulaGraph, then surfaces neighbour queries, mutual connections, friend-of-friend suggestions, and super-node protection as typed C# async methods.

Read

What Belongs in Long-Term Memory

The EF Core Entity and Migration

HNSW Parameter Guidance

The Embedding Service

Chunking Before Storage

Sequential Scan vs. HNSW: Why It Matters

The Recall Query

The SessionStart Injection Hook

What's Next

Sources

Keep reading

Building an Agentic System in .NET, Part 2 — Writing an MCP Server in C#

Building an Agentic System in .NET, Part 1 — Anatomy of an Agent Harness

Graph-Native Data Structures in C#, Part 4 — Graphs & Adjacency with a Social Follow Network