Why Colocation Fails, and Separating the Clusters

In one line: this lesson reverses two decisions from the previous ones — colocation and deterministic recomputation — and the reasoning for both is the same: doing the same expensive work in several places is waste, and mixing two heavy workloads makes both worse.

The two problems

Colocation fails for two independent reasons, and the second is worse

Contention is the obvious one: indexing is CPU- and memory-hungry batch work; search is latency-sensitive interactive work. Run them together and an indexing pass spikes search latency at exactly the moment users notice — and the page cache holding hot index data gets evicted by the indexer's working set.

The inability to scale independently is the deeper problem. Lesson 1 established that these two workloads scale with completely different quantities: search scales with query volume, indexing with corpus volume. Those move independently — a viral week triples queries without adding documents; a big crawl triples documents without adding queries.

Colocated, you must provision for the maximum of both on every node, and you over-provision one to satisfy the other permanently. Separated, you scale each on its own signal.

Contention is a performance problem you could partly tune away. Coupled scaling is a structural one you cannot.

Recomputation is the cost of Lesson 9's elegance

Lesson 9's deterministic replication was genuinely elegant: replicas agree because they process the same documents in the same order, so no data transfer is needed.

The economics are wrong. Index construction is "a heavy pipeline involving hundreds of operations" — running it three times to produce three identical files is 3x the CPU for zero additional information.

That is a good general check to apply: when replicas do identical deterministic work, ask whether it is cheaper to do it once and copy the result. It usually is, because bytes are cheaper to move than computation is to repeat — and that gap has only widened as networks got faster.

The same trade appears elsewhere: materializing a view once versus recomputing per reader, caching a render versus re-rendering. Compute-once-and-distribute beats recompute-everywhere whenever the output is smaller than the work.

The first fix, and its limits

Instead of recomputing the index on every replica, the system computes the inverted index once on the primary node. The resulting index file is then distributed to the replicas. This reduces CPU and memory usage by avoiding redundant computation.

"What are the disadvantages of this solution?"

As the inverted index is transferred to the replicas, this introduces transmission latency when copying the index file, because the index can be very large.

When the primary node receives new indexing operations, the inverted index file changes. Each replica needs to fetch the latest version of the file after a certain number of indexing operations reaches a defined threshold.

This trades CPU for network — and introduces a freshness threshold

The fix converts redundant computation into data transfer, which is the right direction. But it creates two new concerns.

Transfer latency. Index files are large, so pushing them to replicas takes time — during which replicas serve the previous index.

A refresh threshold. Replicas cannot fetch after every indexing operation; they fetch after a threshold number of operations. That threshold is a direct freshness-versus-cost dial: fetch often and pay constant transfer; fetch rarely and serve staler results.

Notice this is the same structure as distributed caching's TTL and pub-sub's retention — a knob trading currency of data against cost of maintaining it. And it is tolerable here for the reason Lesson 1 established: search results are allowed to be slightly stale.

The redesign: three clusters

Modern cloud infrastructure provides high network bandwidth and scalable distributed storage. This enables indexing and search to run on separate clusters without introducing substantial latency overhead. Isolation prevents indexing workloads from degrading search performance, and vice versa. Index files can be replicated across nodes, eliminating the need for recomputation.

The workflow, with N indexing nodes each processing a partition:

When a user submits a query, a Merger node broadcasts it to all searcher nodes. Each searcher generates a response based on its local index segment, and the Merger aggregates these results. As new documents are indexed, search nodes fetch the updated index files.

Distributed storage as the interface is what makes the separation clean

The indexer never talks to the searcher. It uploads a file; searchers download it. That is the entire contract.

Three things fall out, and they are why this design is better than a direct primary-to-replica push:

Backup for free. "If a hardware failure occurs, a replacement node can retrieve the data from this storage." A new node bootstraps by downloading, with no coordination with any peer.

Fan-out for free. One upload serves any number of searchers. The indexer's cost does not grow with the number of searcher replicas — unlike the primary-push version, where the primary pays per replica.

Independent lifecycles. Either cluster can be restarted, resized, or redeployed without the other noticing.

This is exactly pub-sub's argument about getting coordination off the data path, and Lesson 2's choice of a blob store paying off: index files are large, immutable, write-once-read-many binaries — precisely what a blob store is for.

'Without introducing substantial latency overhead' depends on an assumption worth naming

The justification for separating is "modern cloud infrastructure provides high network bandwidth and scalable distributed storage."

That is doing real work. On slower networks, colocation's data locality would matter more and this redesign would be less clearly right. The conclusion is conditional on the environment, not universal.

It also explains why searchers cache frequently queried data in RAM and keep an optional local index: the download happens once, off the query path, and queries are then served from local memory. The network cost is paid at index-refresh time, not per query.

Separation is affordable precisely because the transfer is off the critical path — the same reason offline indexing works at all.

The separation is the same instinct as splitting fetch from parse in a crawler: two workloads with different profiles and different scaling signals should not share a machine, because the bursty one will always be stealing from the latency-sensitive one.

Interview signal by level

Next: how the index actually gets built.