Indexing, Freshness and Serving

In one line: the index is built by a pipeline that runs offline, and almost every serving property you care about is decided by how that pipeline handles updates.

The document pipeline

Every document passes through the same stages, and the analysis stage is the one that must mirror query understanding exactly.

The enrichment stage is where most of the cost lives — running a classifier and an embedding model over every document — and it is the stage that must re-run when a model changes. That is the practical reason the embedding model is a one-way door: changing it means re-running enrichment over the entire corpus.

Segments, and why deletes are cheap

An inverted index is expensive to modify in place. Inserting a document means inserting into many postings lists, each of which is a compressed sorted structure.

So indexes are built from immutable segments. New documents accumulate in a small new segment; a search queries every segment and merges the results. Periodically, small segments are merged into larger ones in the background.

Deletes are the elegant part: a deleted document is marked in a bitmap and filtered from results, and the space is reclaimed only when its segment is next merged. That makes a delete nearly free at write time and slightly expensive at read time.

Two consequences worth knowing:

Merges cost I/O and CPU in bursts, so a system under heavy write load has periodic latency bumps that are not explained by query volume. If a design mentions unpredictable p99, this is a candidate cause.

A document is searchable when its segment is committed, not when it was written. Commit interval is therefore the knob that trades write throughput against freshness, and it is the single number to name when asked how fast an update becomes visible.

The freshness spectrum

Different content needs different paths, and running everything through the slowest one is the common mistake.

The first row is the important one for e-commerce and the one candidates miss. Stock and price change constantly and do not require re-analysing text — they are numeric fields used for filtering and ranking. Updating them separately from the text index means an availability change lands in seconds rather than waiting for a full document re-index.

That split is what makes "availability is part of relevance" affordable.

Sharding and the tail

Past a single machine, the index is sharded by document, and a query is scattered to every shard and gathered.

The property that follows is the one to state: the response time is the slowest shard's response time, not the average. With enough shards, the probability that at least one is slow on any given request approaches certainty, so the tail latency of the whole system is much worse than the tail of any individual shard.

Three standard mitigations: keep the shard count no higher than it needs to be; replicate and send the request to two replicas, taking the first response; and set a deadline after which the gather proceeds with whatever arrived, accepting slightly degraded recall in exchange for a bounded response time.

That last one is a real trade to name — a search that returns 95% of the results in 80ms is usually better than one that returns all of them in 400ms.

Caching

Search caches well because the query distribution is skewed, and there are three distinct layers.

Result cache — the full response for a query. Highest value, and it only works for queries whose results do not depend on the user. Personalised or permissioned results cannot use it, and a cache key that omits the identity on a permissioned corpus is a data leak.

Postings cache — hot terms' postings lists kept in memory. Shared across all queries containing that term, so it helps the tail as well as the head, which the result cache does not.

Filter cache — the document set matching a common filter, as a bitmap. "In stock" and "free shipping" are evaluated on most queries and change slowly.

The middle one is the most valuable and least discussed: it is the only layer that speeds up queries you have never seen.

Invalidation is a product decision by vertical. A news search cannot serve a five-minute-old result cache; a documentation search can serve an hour. And a cached result must be invalidated by inventory changes, which is another reason the volatile fields are separated — a price change should invalidate a cached page without invalidating the postings.

The budget

Where a search request's time goes, roughly in order:

Query understanding is small — microseconds to a few milliseconds, unless an LLM rewrite is on the path, which is why that belongs offline for head queries. Retrieval is the scatter-gather, so it is shard-tail bound. Ranking is the model over surviving candidates, and its cost is candidates times feature computation. Facet aggregation is frequently the largest single item and the one people forget to budget. Assembly and snippet generation are small but not free — snippets require fetching document text, which is a second round trip if the text is not in the index.

The practical ordering advice: compute facets in parallel with ranking rather than after it, since they depend on the matching set and not on the order.

Next: the whole thing, as an interview.