Idempotent Bulk Ingestion into NebulaGraph from .NET: Solving the Duplicate Problem at the VID Level

Why 'Duplicates' Mean Something Different in a Graph DB

If you come from a relational background, or from the ArangoDB migration covered earlier, your first instinct for duplicate prevention is a unique index or an ON CONFLICT DO NOTHING clause. NebulaGraph has no unique index to lean on, and that difference is behind a lot of ingestion bugs.

In NebulaGraph, vertices are keyed by VID (vertex identifier) and edges are keyed by (src_vid, dst_vid, edge_type, rank). There are no uniqueness constraints apart from these keys. A plain INSERT VERTEX behaves like a key-value PUT: if you insert a vertex with an existing VID and tag, the later write overwrites the earlier one. INSERT VERTEX IF NOT EXISTS and INSERT EDGE IF NOT EXISTS do exist, but they compare only the key (VID and tag, or the edge key), not the properties, and the documentation warns that the extra read has a significant cost.

So idempotency is built into the storage model, but only if your VIDs are deterministic. Non-deterministic VIDs (UUIDs generated at insert time, counters handed out by the loader) create genuine duplicates, because each write lands on a different key.

The fix is architectural, not operational. You don't add a dedup step after ingestion; you design the VID scheme so the same logical entity always resolves to the same VID, which makes repeated inserts safe by construction.

The Golden Rule: Deterministic VIDs from Business Keys

The VID type, FIXED_STRING(<N>) or INT64, is fixed at CREATE SPACE time and cannot be changed later. A good fit for bulk ingestion from .NET is a string VID derived by hashing the natural business key with FNV-1a. FNV-1a is fast, deterministic and trivial to implement without an extra dependency, and the result always has the same length.

public static class VidGenerator
{
    // FNV-1a 64-bit: the same business key always gives the same 16-character VID
    private const ulong FnvPrime = 0x00000100000001B3UL;
    private const ulong FnvOffset = 0xCBF29CE484222325UL;
 
    public static string FromBusinessKey(string businessKey)
    {
        ulong hash = FnvOffset;
        foreach (byte b in Encoding.UTF8.GetBytes(businessKey))
        {
            hash ^= b;
            hash *= FnvPrime;
        }
        // 16 hex characters, so the space needs vid_type = FIXED_STRING(16) or wider
        return hash.ToString("x16");
    }
 
    // Composite keys: namespace the input so "user:42" != "order:42"
    public static string FromCompositeKey(string type, string id)
        => FromBusinessKey($"{type}:{id}");
}

Why FNV-1a and not GetHashCode() or MD5? string.GetHashCode() is randomized per process in .NET, so the same key gets a different value after every restart. MD5 works, but it's more than you need. FNV-1a is deterministic and runs in a tight loop.

Partition spread is not the reason to hash. NebulaGraph does that itself: a string VID is hashed with MurmurHash2, and the partition is vid % numParts + 1. Integer VIDs go through the same modulo, so sequential integers are spread round robin across partitions. What hashing gives you is a key with a fixed length that fits the space's FIXED_STRING(N), whatever the length of the business key.

Edge Duplication and the Role of Rank

Vertices are the easy part. Edges are where silent duplication tends to sneak in.

An edge in NebulaGraph is uniquely identified by (src_vid, dst_vid, edge_type, rank). The default rank is 0. If your import pipeline sets rank to a timestamp or an auto-increment counter, every import run creates a new edge, even though the business relationship has not changed. The graph accumulates phantom edges that look correct to a casual query but inflate degree counts and traversal results.

The fix: pin rank to 0 for relationships that have no natural ordering dimension, or compute a deterministic rank from the business relationship key:

# nGQL: idempotent edge insert with explicit rank 0
INSERT EDGE FOLLOWS () VALUES "a1b2c3d4e5f60718"->"e5f6a7b8c9d00112"@0:();
 
# If rank carries business meaning (e.g., version number), derive it deterministically
INSERT EDGE ORDERED_BY () VALUES "a1b2c3d4e5f60718"->"e5f6a7b8c9d00112"@42:();

In .NET, this means your edge-building code must never use DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() as a rank. Decide at schema design time: either rank is always 0, or it is computed from an immutable attribute of the relationship (invoice sequence number, order index, and so on).

Choosing an Import Path at Scale

For a .NET team owning the data pipeline end-to-end, driver-side batched INSERT is the natural starting point. For very large datasets produced by an ETL process, export to CSV and let NebulaGraph Importer handle it. It supports S3, GCS, HDFS and SFTP sources, and handles retry and load balancing across graph hosts internally. When you're moving hundreds of millions of records and storage compute is a concern, Nebula Exchange's SST bulk ingest moves that work to the Spark cluster.

Note: Exchange 3.4.0 shipped without Kafka and Pulsar support. Current Exchange (3.8.0) reads from both, so a streaming source only needs you to avoid that one release.

A .NET Idempotent Ingestion Pipeline

The driver-side pattern: stream the source, batch into fixed-size chunks, send each batch as one nGQL statement, checkpoint completed batch offsets so a failed run resumes, and retry transient failures with exponential backoff.

public sealed class NebulaIngestionPipeline(INebulaClient client, ICheckpointStore checkpoint)
{
    private const int BatchSize = 256;
 
    public async Task IngestVerticesAsync(
        IAsyncEnumerable<UserRecord> source,
        CancellationToken ct)
    {
        long batchIndex = await checkpoint.GetLastCompletedBatchAsync();
        long current = 0;
 
        await foreach (var batch in source.Chunk(BatchSize).WithCancellation(ct))
        {
            current++;
            if (current <= batchIndex)
                continue; // skip already-committed batches on resume
 
            var ngql = BuildVertexInsert(batch);
 
            await Policy
                .Handle<NebulaTransientException>()
                .WaitAndRetryAsync(3, attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt)))
                .ExecuteAsync(() => client.ExecuteAsync(ngql, ct));
 
            await checkpoint.CommitAsync(current);
        }
    }
 
    private static string BuildVertexInsert(UserRecord[] batch)
    {
        // Build a multi-value INSERT for the entire batch
        var values = string.Join(",\n",
            batch.Select(r =>
            {
                var vid = VidGenerator.FromCompositeKey("user", r.UserId);
                return $"\"{vid}\":({Quote(r.Name)}, {r.Age})";
            }));
 
        return $"""
            INSERT VERTEX Person(name, age) VALUES
            {values};
            """;
    }
 
    // nGQL string literal: escape the backslash first, then the double quote
    private static string Quote(string s)
        => "\"" + s.Replace("\\", "\\\\").Replace("\"", "\\\"") + "\"";
}

INebulaClient, ICheckpointStore and NebulaTransientException are your own small abstractions: the client wraps a NebulaNet NebulaConnection, and the checkpoint store can be a single row in your own database. NebulaNet's ExecuteAsync throws a plain Exception whose message ends in "execute failed." for any failed statement, a syntax error as much as a timeout, so the wrapper should turn only the retryable ones into NebulaTransientException. Otherwise Polly retries a syntax error three times. source.Chunk on an IAsyncEnumerable comes with .NET 10's built-in System.Linq.AsyncEnumerable.

Because VIDs are deterministic, replaying any batch is safe: re-inserting a vertex with an existing VID overwrites it with the same data. The checkpoint exists only for efficiency (skip already-sent batches), not for correctness. After a crash you don't need to reconcile partial state; you resume from the last committed batch.

Be disciplined about escaping. NebulaGraph's graph service does have an executeWithParameter call, but the NebulaNet client's NebulaConnection exposes only ExecuteAsync(session, statement), so with it you build nGQL strings by hand. Wrap every string property in double quotes and escape the backslash before the quote, as Quote does above; a name like O'Brien "Bob" then goes through intact.

INSERT vs UPSERT: When to Use Each

Use INSERT (overwrite-by-key) for bulk loads. UPSERT in NebulaGraph is, in the documentation's words, a read-modify-write serialization operation at the partition level. It reads the current state, applies the mutation and writes back, one operation at a time per partition. At bulk-load volume that serialization becomes the bottleneck.

UPSERT earns its place in incremental update scenarios where you genuinely need conditional semantics:

# Conditional UPSERT: only update if score is still below 100
UPSERT VERTEX ON Player "a1b2c3d4e5f60718"
    SET score = score + 10
    WHEN score < 100
    YIELD score AS newScore;

For counters, max-wins merges, or partial property updates during live operation: UPSERT. For a bulk load pipeline processing millions of records: INSERT.

Pre-Import Deduplication of the Source

Deterministic VIDs handle re-import idempotency, but they do not handle the case where your source data itself contains two rows that represent the same business entity with conflicting property values. Both rows hash to the same VID; the second write wins, which may not be the right outcome.

Before ingestion, group source rows by the business key and apply a deterministic merge strategy. Last write wins by source timestamp is usually fine, but document it explicitly. In LINQ:

var deduped = sourceRows
    .GroupBy(r => r.BusinessKey)
    .Select(g => g.OrderByDescending(r => r.UpdatedAt).First())
    .ToList();

For very large sources that don't fit in memory, do this deduplication in the ETL stage (SQL DISTINCT ON, Spark dropDuplicates("business_key")) before writing the CSV or streaming records to the .NET pipeline.

Verifying Integrity After Load

After a full or incremental load, run a stats job against your graph space and compare its vertex and edge counts with COUNT(DISTINCT business_key) from your source:

# Trigger async stats collection
USE my_graph;
SUBMIT JOB STATS;
 
# Poll until the job shows FINISHED, then read the counts
SHOW JOB <job_id>;
SHOW STATS;

SHOW STATS returns per-tag vertex counts, per-edge-type edge counts and the space totals from the last finished stats job. If the counts match your source distinct counts, your VID scheme is working. If edge counts exceed the distinct relationships in your source, something is generating non-deterministic ranks and creating extra edges. If vertex counts are lower, you likely have a key collision in the hash function (exceedingly rare with FNV-1a 64-bit, but worth checking with a quick collision test on your key space).

Also run SHOW HOSTS to confirm all storage nodes are ONLINE before you trust the stats, because a node that went offline mid-import will skew the counts.

Gotchas Worth Calling Out

Super-nodes: A vertex with millions of edges keeps its outgoing edges on its own partition. Traversals touching a super-node become single-partition read hotspots. Design your schema to avoid super-nodes, or apply degree limits in queries (LIMIT on GO traversals).

Dangling edges: NebulaGraph allows inserting edges before their source or destination vertices exist. Writes are not atomic across vertices and edges, so a batch that fails halfway can leave dangling edges. Since INSERT is idempotent, the correct response is always to retry the full batch, not to attempt a surgical rollback.

Keyword tag names: USER is a non-reserved keyword in nGQL. In our test NebulaGraph accepted CREATE TAG User without backticks but stored the tag as user, so SHOW STATS listed user and code looking for User found nothing. Pick tag and edge names that are not keywords (this article uses Person), or wrap them in backticks.

Business key stability: Idempotency breaks the moment a business key changes. If your source system reassigns natural keys (rare, but it happens with legacy RDBMS migrations), your VID scheme will generate a new vertex for what is conceptually the same entity, and the old vertex becomes an orphan. Audit key stability before committing to a VID strategy.

UPSERT at scale. Once more: bulk UPSERT is a trap. Measure throughput with a small batch before committing to it for a million-row load.

The code in this article was run against NebulaGraph 3.8.0 (the current Community Edition) with the NebulaNet 3.0.0 client on .NET 10.