Building an Agentic System in .NET, Part 4: Keeping Memory True

The Hard Part Is Not Storing Memory, It Is Keeping It True

Here is something that happened on a real team. An agent was managing infrastructure tasks. Its memory store held a fact: "The server is fronted by aaPanel nginx." That was true on the day it was written. Three weeks later the team moved to a containerised reverse proxy, and nobody updated the agent's memory. The agent kept producing nginx config snippets, reload commands aimed at a process that was gone, and health check paths that no longer existed. The memory was not corrupted. It was just old. It had turned into a lie that looked exactly like a fact.

That is the problem this part is about. Parts 1 to 3 built the agent skeleton, the tool loop and the RAG pipeline. Now the memory layer has to become something you can act on.

What Every Memory Record Must Carry

Semantic Kernel's Vector Store abstractions, which reached GA for .NET earlier this year, give you no timestamp, no provenance and no TTL field. There is no decay mechanism in the framework. Everything in this section is on you.

The smallest record that works looks like this:

public class AgentMemoryRecord
{
    [VectorStoreRecordKey]
    public string Id { get; set; } = Ulid.NewUlid().ToString();
 
    [VectorStoreRecordData]
    public string Content { get; set; } = string.Empty;
 
    [VectorStoreRecordData]
    public string Source { get; set; } = string.Empty; // tool name, user id, document URI
 
    [VectorStoreRecordData]
    public DateTimeOffset WrittenAt { get; set; } = DateTimeOffset.UtcNow;
 
    [VectorStoreRecordData]
    public double DecayScore { get; set; } = 1.0; // 1.0 = fresh, approaches 0 over time
 
    [VectorStoreRecordData]
    public bool Superseded { get; set; } = false;
 
    [VectorStoreRecordVector(Dimensions: 1536)]
    public ReadOnlyMemory<float> Embedding { get; set; }
}

WrittenAt is your audit trail. Source is provenance, so an agent that recalls a fact can answer where did this come from before it acts. DecayScore is maintained by the decay worker below. Superseded is set by the contradiction detector, so pruning can find stale records without a vector scan.

One warning. If you are migrating from the older IMemoryStore API, it is officially deprecated. Do not copy samples from repositories written before 2025 that use ISemanticTextMemory. It still compiles, and that is the trap.

Decay and Deduplication as Hosted Services

The decay worker and the deduplication worker are both long running background processes. In .NET 10 the right primitive is BackgroundService from Microsoft.Extensions.Hosting.

Two rules matter here:

1. These services are singletons. Never inject a scoped dependency such as an EF Core DbContext straight into one. Use IServiceScopeFactory and create a fresh scope per work cycle, otherwise you get the captive dependency bug.
2. Do not run a synchronous scan of the whole collection before the first await in ExecuteAsync. The host waits until ExecuteAsync yields, so a cold dedup pass over a big vector store delays startup for the entire application.

public class MemoryDecayWorker : BackgroundService
{
    private readonly IServiceScopeFactory _scopeFactory;
    private readonly ILogger<MemoryDecayWorker> _logger;
    private static readonly TimeSpan _interval = TimeSpan.FromHours(6);
    private const double HalfLifeDays = 14.0;
 
    public MemoryDecayWorker(IServiceScopeFactory scopeFactory,
        ILogger<MemoryDecayWorker> logger)
    {
        _scopeFactory = scopeFactory;
        _logger = logger;
    }
 
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        // Yield immediately so host startup is not blocked
        await Task.Yield();
 
        using var timer = new PeriodicTimer(_interval);
        while (await timer.WaitForNextTickAsync(stoppingToken))
        {
            try
            {
                await RunDecayCycleAsync(stoppingToken);
            }
            catch (Exception ex) when (ex is not OperationCanceledException)
            {
                _logger.LogError(ex, "Decay cycle failed; will retry on next tick");
            }
        }
    }
 
    private async Task RunDecayCycleAsync(CancellationToken ct)
    {
        await using var scope = _scopeFactory.CreateAsyncScope();
        var collection = scope.ServiceProvider
            .GetRequiredService<IVectorStoreRecordCollection<string, AgentMemoryRecord>>();
 
        await collection.EnsureCollectionExistsAsync(ct);
 
        var now = DateTimeOffset.UtcNow;
        // Paginate to avoid loading the entire collection into memory
        var options = new GetRecordOptions { Top = 500, Skip = 0 };
        await foreach (var record in collection.GetAsync(_ => !_.Superseded, options, ct))
        {
            var ageDays = (now - record.WrittenAt).TotalDays;
            // Exponential decay: score = e^(-lambda * age)
            var lambda = Math.Log(2) / HalfLifeDays;
            record.DecayScore = Math.Exp(-lambda * ageDays);
            await collection.UpsertAsync(record, cancellationToken: ct);
        }
    }
}

Note EnsureCollectionExistsAsync. It replaced CreateCollectionIfNotExistsAsync in the May 2025 GA review. The old name throws at runtime on the renamed connectors, and the compiler will not catch it when the call goes through an interface.

PeriodicTimer has been there since .NET 6 and does not accumulate drift, which makes it a better choice than a raw Task.Delay loop and costs you no extra package. The half life should be configurable. Fourteen days is a sensible default for infrastructure facts. Business rules you learned once might deserve 90 days or more.

Deduplication: The Threshold That Must Not Merge Different Facts

The deduplication worker embeds a candidate memory, searches for near neighbours, and merges records that mean the same thing. The setting that decides whether this helps or hurts is the similarity threshold.

Above 0.97 cosine similarity you are safe. That is the same fact written twice in slightly different words. Below 0.95 you start merging things that are related but not the same. "nginx on port 80" and "nginx acting as TLS terminator" might sit at 0.94 and still describe two different operational truths. Get this number wrong and you delete information without any sign that you did.

For contradiction detection, do something narrower. When a new memory arrives, embed it, take the top five neighbours above 0.90, and compare the structured fields: host name, port, protocol, flag name. If an older record names the same entity with a different value, set Superseded = true on the older one and log a contradiction event for a human to look at. Do not overwrite silently. The audit trail is the whole point.

Verification on Read for Facts You Act On

A decay score tells you a memory is probably old. When a memory names a file path, a feature flag or an endpoint, probably is not good enough. You need a live check.

The place to put it is after collection.SearchAsync(...) returns and before you build the prompt context. This helper adds a staleness banner and, when it matters, fires a probe:

public static class MemoryContextBuilder
{
    private const double StaleThreshold = 0.35;
    private const int StaleAgeDays = 7;
 
    public static async Task<string> BuildContextAsync(
        IEnumerable<VectorSearchResult<AgentMemoryRecord>> results,
        ILiveProbeService probeService,
        CancellationToken ct = default)
    {
        var sb = new StringBuilder();
        var now = DateTimeOffset.UtcNow;
 
        foreach (var result in results)
        {
            var record = result.Record;
            var ageDays = (now - record.WrittenAt).TotalDays;
            var isStale = record.DecayScore < StaleThreshold
                          || ageDays > StaleAgeDays;
 
            sb.AppendLine($"[Memory | Source: {record.Source} | Written: {record.WrittenAt:yyyy-MM-dd}]");
 
            if (isStale)
            {
                sb.AppendLine(
                    $"[STALE: written {ageDays:F0} days ago, verify before acting]");
            }
 
            sb.AppendLine(record.Content);
 
            // For memories referencing endpoints or file paths, fire a live probe
            if (ContainsActionableReference(record.Content) && isStale)
            {
                var probeResult = await probeService.ProbeAsync(record.Content, ct);
                if (!probeResult.IsAlive)
                    sb.AppendLine(
                        $"[PROBE FAILED: {probeResult.Reason}, do not act on this memory]");
            }
 
            sb.AppendLine();
        }
 
        return sb.ToString();
    }
 
    private static bool ContainsActionableReference(string content) =>
        content.Contains("http", StringComparison.OrdinalIgnoreCase)
        || content.Contains("/var/", StringComparison.OrdinalIgnoreCase)
        || content.Contains(":443", StringComparison.OrdinalIgnoreCase)
        || content.Contains(":80", StringComparison.OrdinalIgnoreCase);
}

Treat that banner as part of the prompt, not as decoration. The model reads it, and if your system prompt tells it to respect the banner, it will either ask for confirmation or refuse to run an infrastructure changing tool until the fact is verified. This is also where the memory layer meets the human in the loop hooks that the Microsoft Agent Framework's checkpointing supports.

Scheduled Pruning

Marking a record Superseded is only half the job. You still need a pass that removes it, or the collection grows forever and similarity search gets worse with it. A small BackgroundService on a daily PeriodicTimer is enough: query for Superseded == true or DecayScore < 0.05, delete in batches, log a summary. Quartz.NET or Hangfire are worth adding only if you need cron style schedules or a dashboard. For plain periodic work, PeriodicTimer with no extra dependency is cleaner.

Practical Gotchas Before You Ship

• The legacy IMemoryStore still compiles. If someone copies a 2024 sample, they quietly end up on the deprecated path. Add an architecture test that fails if any type in the solution touches IMemoryStore.
• BackgroundServiceExceptionBehavior in .NET 6 and later does not stop the host when ExecuteAsync throws. Wrap your inner loop in try/catch yourself, and never let the decay worker die quietly.
• Agent Framework memory connectors (Microsoft.SemanticKernel.Connectors.InMemory was at 1.66.0-preview in February 2026) can still ship breaking changes. Pin the versions and run the full memory integration suite on every update.
• Registration order matters. Hosted services start in order. If your vector store connection is itself a hosted service, such as a connection pool initialiser, register it before the decay and dedup workers.

Closing Thought

The aaPanel incident at the top of this article was not a reasoning bug. The agent reasoned correctly from what it knew. The problem was that what it knew was wrong, and nothing in the system told it so. Timestamps, decay scores, contradiction detection and verification on read are not things you bolt on after an incident. They are the baseline for any memory backed system that takes real actions. Build them first.