Operating the Index

In one line: building an index is a day's work and running one through a year of updates, deletes and model changes is where the design decisions actually get tested.

Deletes are the sharp edge

Graph indexes have no clean removal. Unlinking a node would mean rewiring its neighbours and potentially repairing paths that routed through it, which is expensive and can disconnect regions.

So deletion is a tombstone: the node stays in the graph, participates in traversal, and is filtered out of results at the end.

The consequence to state: a high-churn corpus degrades silently. Search latency is unchanged, the index is not obviously broken, and quality erodes as the fraction of tombstones grows.

The countermeasures are unglamorous and necessary. Track the tombstone ratio as a monitored metric. Compact or rebuild when it crosses a threshold — something like 10–20% depending on tolerance. And when estimating capacity, remember a deleted vector still occupies memory until compaction.

Freshness: the two-tier pattern

Building a graph index over a large corpus takes hours, so you cannot rebuild on every change. The standard resolution splits the index by age.

Every query hits both and merges. The delta stays small enough that even brute force over it is fast, so it needs no sophisticated structure.

Two details worth naming. Deletes must be applied to both tiers, or a document deleted from main but present in delta reappears. And the merge has the same score-comparability problem as hybrid search — if the two tiers are scored independently, rank-based fusion is safer than comparing raw distances.

Sharding

Past the point where one index fits comfortably, you split. There are two ways and they behave very differently.

Shard by attribute wherever the access pattern allows it. A query touching one shard avoids fan-out entirely, and — as the filtering lesson argued — it converts the most common filter into a routing decision.

The cost is imbalance. In a B2B product, one enterprise customer may hold more documents than a thousand small ones, so shards need occasional splitting and the largest tenant sets your per-shard sizing.

Random sharding balances perfectly and makes every query a fan-out, where total latency is the slowest shard's latency. With ten shards you are exposed to the ninetieth-percentile tail on every single query — the standard tail-amplification problem, arriving here as it does everywhere else.

Multi-tenancy

Three arrangements, and the choice is as much about compliance as performance.

Shared index with a tenant filter. Cheapest, and it inherits the filtering problem exactly — and if the filter fails, one tenant sees another's data. A high-consequence bug behind a low-visibility mechanism.

Index per tenant. Strong isolation, no filter, easy per-tenant deletion for data-residency and right-to-erasure requests. The cost is many small indexes, which is memory overhead per index and an operational burden at thousands of tenants.

Hybrid. Dedicated indexes for large tenants, a shared filtered index for the long tail. Usually the right answer at scale, and worth proposing as such rather than as a compromise.

Re-embedding is a migration

The scenario from the model-choice lesson, now as an operation. Changing the embedding model invalidates every vector, and the corpus must be re-embedded and re-indexed while the system stays up.

The shadow step is the one that pays for itself. Retrieval quality on the new model is genuinely uncertain until you see real queries against real data, and comparing side by side on live traffic is the only honest test — offline evaluation on a sampled set will not surface the query patterns you did not think to include.

Plan for both indexes to exist simultaneously. That means double the memory during the migration, which is a capacity requirement people forget until the week they need it.

Next: choosing what to run all this on.