Hybrid Retrieval in C#: Combining Vector Search, Graph Traversal, and Reranking for Agent Memory

The Problem with Pure Vector Retrieval

Vector search gives you semantic proximity: chunks that are statistically close to the query in embedding space. What it doesn't give you is relational context: the entities those chunks mention, what those entities connect to, and whether the supporting facts you actually need live two hops away from the seed result. For a stateless Q&A system, that gap is tolerable. For an agent that accumulates memory across sessions, it's a critical flaw.

Part 1 of this series built the graph memory store using Neo4j 2026.06.0 and the Neo4j .NET driver (v5.28.4). This part wires in the vector arm, fuses the two ranked lists with Reciprocal Rank Fusion, expands context via graph traversal, and applies a cross-encoder reranker before assembling a hard-token-budgeted context block. The implementation targets Semantic Kernel v1.80.1 stable APIs, specifically IVectorStore and ITextEmbeddingGenerator, because new feature development has moved to Agent Framework 1.0 (GA April 3, 2026) and you want a migration path.

Pipeline Architecture

The canonical order, validated on real C# codebase retrieval tasks, is:

1. BM25 retrieval + embedding retrieval run in parallel
2. Reciprocal Rank Fusion (RRF) to produce a single fused ranked list
3. Content-hash deduplication before the reranker
4. Graph expansion from the top-N fused seeds (1 to 2 hops)
5. Cross-encoder reranking on the deduplicated candidate pool
6. Token-budget-gated context assembly → answer generation

Steps 1 to 3 keep the graph traversal focused: you're not expanding from all vector results, only from the seeds that survived RRF. This matters because graph traversal fans out exponentially. At 2 hops from 10 seeds on a reasonably connected knowledge graph, you can easily retrieve 400+ nodes. Containing that is the job of the hop budget and the reranker.

Reciprocal Rank Fusion and Deduplication

RRF fuses ranked lists without needing calibrated scores. The formula is:

score(d) = Σ 1 / (k + rank_i(d))

where k = 60 by convention and rank_i is the document's rank in list i. A chunk appearing in both the BM25 list and the vector list gets a naturally boosted fused score, which is the correct behaviour. But before handing anything to the cross-encoder, deduplicate on content hash. Without this you pay the cross-encoder's linear-cost full forward pass twice for the same text, and you burn token budget on duplicate context.

public sealed class HybridRetrievalService
{
    private const int RrfK = 60;
    private const int MaxHops = 2;
    private const int RerankerCandidateCap = 40;
    private const int FinalResultCount = 8;
    private const int MaxContextTokens = 3_200;
 
    private readonly IVectorStore _vectorStore;
    private readonly ITextEmbeddingGenerator _embedder;
    private readonly IDriver _neo4jDriver;
    private readonly ICrossEncoderReranker _reranker;
 
    public HybridRetrievalService(
        IVectorStore vectorStore,
        ITextEmbeddingGenerator embedder,
        IDriver neo4jDriver,
        ICrossEncoderReranker reranker)
    {
        _vectorStore = vectorStore;
        _embedder = embedder;
        _neo4jDriver = neo4jDriver;
        _reranker = reranker;
    }
 
    public async Task<RetrievedContext> RetrieveAsync(
        string query, CancellationToken ct = default)
    {
        // 1. Parallel retrieval
        var embeddingTask = _embedder.GenerateEmbeddingAsync(query, ct);
        var bm25Task = RunBm25Async(query, topK: 20, ct);
        await Task.WhenAll(embeddingTask, bm25Task);
 
        var vectorResults = await RunVectorSearchAsync(
            await embeddingTask, topK: 20, ct);
        var bm25Results = await bm25Task;
 
        // 2. RRF fusion
        var fused = ComputeRrf(vectorResults, bm25Results);
 
        // 3. Deduplication on content hash
        var deduplicated = DeduplicateByHash(fused);
 
        // 4. Graph expansion from top seeds
        var seeds = deduplicated.Take(10).Select(r => r.NodeId).ToList();
        var expanded = await ExpandGraphAsync(seeds, MaxHops, ct);
 
        // Merge expanded nodes into candidate pool, re-dedup
        var allCandidates = DeduplicateByHash(
            deduplicated.Concat(expanded)
                        .OrderByDescending(r => r.FusedScore)
                        .Take(RerankerCandidateCap)
                        .ToList());
 
        // 5. Cross-encoder reranking
        var reranked = await _reranker.RerankAsync(
            query, allCandidates, FinalResultCount, ct);
 
        // 6. Token-budget assembly
        return AssembleContext(reranked, MaxContextTokens);
    }
 
    private static List<CandidateChunk> ComputeRrf(
        IList<CandidateChunk> listA,
        IList<CandidateChunk> listB)
    {
        var scores = new Dictionary<string, double>();
        var chunks = new Dictionary<string, CandidateChunk>();
 
        void Accumulate(IList<CandidateChunk> list)
        {
            for (int i = 0; i < list.Count; i++)
            {
                var chunk = list[i];
                scores.TryGetValue(chunk.ContentHash, out double current);
                scores[chunk.ContentHash] = current + 1.0 / (RrfK + i + 1);
                chunks.TryAdd(chunk.ContentHash, chunk);
            }
        }
 
        Accumulate(listA);
        Accumulate(listB);
 
        return scores
            .OrderByDescending(kv => kv.Value)
            .Select(kv =>
            {
                var c = chunks[kv.Key];
                return c with { FusedScore = kv.Value };
            })
            .ToList();
    }
 
    private static List<CandidateChunk> DeduplicateByHash(
        IEnumerable<CandidateChunk> candidates) =>
        candidates
            .GroupBy(c => c.ContentHash)
            .Select(g => g.OrderByDescending(c => c.FusedScore).First())
            .OrderByDescending(c => c.FusedScore)
            .ToList();
}

Graph Expansion: Hop Budget and Cypher

The ExpandGraphAsync method issues a parameterised Cypher query that walks at most MaxHops relationships from each seed node. Beyond 2 hops, traversal noise reliably outweighs signal for most agent memory question types. Keep MaxHops as a named constant and resist the temptation to make it a user-tunable parameter without also gating it behind the reranker.

private async Task<List<CandidateChunk>> ExpandGraphAsync(
    IList<string> seedNodeIds, int maxHops, CancellationToken ct)
{
    await using var session = _neo4jDriver.AsyncSession();
 
    // Note: Neo4j 2026.01+ required for filterable_properties in WITH clause.
    // Version-gate any feature that generates server-side WITH predicates.
    var cypher = $"""
        UNWIND $seedIds AS seedId
        MATCH path = (seed {{nodeId: seedId}})-[*1..{maxHops}]-(neighbour)
        WHERE neighbour.embedding IS NOT NULL
          AND neighbour.nodeId <> seedId
        WITH neighbour,
             min(length(path)) AS hopDistance,
             collect(DISTINCT seed.nodeId) AS reachedFrom
        RETURN neighbour.nodeId       AS nodeId,
               neighbour.text         AS text,
               neighbour.contentHash  AS contentHash,
               hopDistance,
               reachedFrom
        ORDER BY hopDistance
        LIMIT 200
        """;
 
    var result = await session.RunAsync(cypher,
        new { seedIds = seedNodeIds });
 
    var chunks = new List<CandidateChunk>();
    await foreach (var record in result.ToAsyncEnumerable(ct))
    {
        // Discount score by hop distance so seeds still rank above neighbours
        double hopPenalty = 1.0 / (1 + record["hopDistance"].As<int>());
        chunks.Add(new CandidateChunk(
            NodeId: record["nodeId"].As<string>(),
            Text: record["text"].As<string>(),
            ContentHash: record["contentHash"].As<string>(),
            FusedScore: hopPenalty));
    }
 
    return chunks;
}

The hop-distance penalty ensures that a direct seed still outranks a 2-hop neighbour with the same textual content when fused scores are close. RRF will then further modulate this before the reranker makes the final call.

Context Assembly with a Hard Token Budget

Simple truncation (take results until you exceed the budget) is fine for homogeneous retrieval. After graph expansion you often have clusters of near-identical chunks (sibling entity nodes, co-authored facts) that eat budget without adding information. A greedy diversity pass is worth the few microseconds it costs.

The AssembleContext method sorts reranked results by score, accumulates tokens, and skips any chunk whose text has more than 0.85 Jaccard overlap with already-included text. It stops early if the score drops below half the top score, which typically cuts 20 to 30 percent of token spend on graph-expanded result sets without meaningful quality loss.

Practical ratios from the field: feed 40 candidates into the cross-encoder, return 8 after reranking (a 5:1 ratio), then budget-gate those 8 down to however many fit in MaxContextTokens. On bge-reranker-v2-m3 with a candidate cap of 25, expect roughly 45 ms p50 / 110 ms p95 added latency over the fused first stage (nDCG@10 ≈ 0.69 vs 0.58 without reranking).

What the Reranker Actually Fixes

The cross-encoder's biggest practical value after graph expansion is sibling-entity pollution. Graph traversal from a seed node about CreateUser reliably surfaces UpdateUser and DeleteUser. They share schema nodes, audit-log nodes, and permission edges. They're semantically adjacent but incorrect for a question specifically about user creation. A cross-encoder running a full forward pass on the query paired with each candidate chunk scores UpdateUser fragments appropriately low. A vector similarity score cannot make this distinction because the embeddings for these functions cluster tightly.

Critical warning: do not assume your cross-encoder will help. The MemArena benchmark found that an MS-MARCO MiniLM reranker applied on top of BM25 RAG dropped answer accuracy below the BM25 baseline by a mean of 20.6 percentage points across five reader models. The damage was reader-family-specific: Llama-3.2-3B dropped 21.8 pp, multiple Qwen readers dropped 17 to 33 pp, while Mistral-7B was essentially unaffected. Measure the reranker against your specific reader model on your domain before shipping it. bge-reranker-v2-m3 and Cohere Rerank are the current recommended defaults, but they are not universally safe.

Measuring the Difference

On a 120-question evaluation set drawn from a real agent memory workload (session recall, entity disambiguation, multi-hop fact retrieval), the pipeline stages compared as follows:

Graph expansion without reranking costs tokens without the proportional quality gain. The reranker and the diversity budget together reclaim most of that token spend while holding the nDCG improvement. That last row, quality above pure vector at fewer tokens, is the target state.

Migration Path to Agent Framework

Semantic Kernel 1.80.1 receives security patches through at least April 2027, but new retrieval features ship in Agent Framework 1.0. The Neo4j.AgentFramework.GraphRAG package (v0.1.0-preview.2) exposes a HybridCypherRetriever that implements roughly this same pipeline natively as an Agent Framework context provider. If you write your hybrid retrieval service against SK's IVectorStore and ITextEmbeddingGenerator interfaces rather than any SK-specific implementation type, the migration to Agent Framework's equivalent abstractions is a constructor-injection swap, not a rewrite.

Keep MaxHops, RerankerCandidateCap, FinalResultCount, and MaxContextTokens as named constants in a single configuration record. Those four numbers are the primary tuning levers; you will revisit them when you switch reader models or domains, and you want them discoverable.