Search

In one line: the dual-index design here is the clearest example in the course of splitting an index by recency rather than by key, and the reason it works is entirely about how people actually search.

The search service indexes roughly a trillion records to serve requests within 100 milliseconds. Twitter uses Apache Lucene for real-time search, utilizing an inverted index. To ensure low latency, a real-time index of recent tweets (from the past week) is stored in RAM. A full index, which is significantly larger, handles historical data via batch processing.

Splitting by recency works because search traffic follows recency

Most index partitioning splits by key — shard by term, by document ID, by hash. This splits by time, and the justification is behavioural rather than technical.

Almost all searches are about now. People search Twitter to find out what is happening — a breaking story, a live event, a trending hashtag. The overwhelming majority of queries are satisfied by tweets from the last few hours, let alone the last week.

So the partition matches the traffic:

Lesson 3's arithmetic makes the RAM half credible:

7 days x 1.5B tweets/day = 10.5 billion tweets
at 250 bytes of text     = ~2.6 TB

A few terabytes of text plus the inverted index over it, spread across a cluster. That is a large but entirely ordinary amount of RAM — and it buys sub-millisecond access for the queries that constitute nearly all traffic.

When access probability is a strong function of a single dimension, partition on that dimension and put the hot partition in fast storage. Recency is that dimension here, in a way it simply is not for Yelp's places or Google's web corpus.

Two indexes means two write paths and a seam between them

The design presents this as one system. It is two, with different update mechanics:

Real-time index: a tweet must be searchable within seconds of posting. So indexing is streaming — the tweet is added to the in-memory index as part of the fan-out from Lesson 5.

Full index: built by batch processing, so a tweet appears there hours later.

That produces a seam, and the seam has to be handled:

Overlap, don't abut. If the real-time index covers exactly 7 days and the batch job runs daily, tweets can fall between them. The windows must overlap, with deduplication at merge.

Queries spanning the boundary hit both. A search for the last ten days must query both indexes and merge — with different latencies, so the response is as slow as the batch side.

Rebuilding the batch index is expensive. A trillion records is not re-indexed casually, which is why it is batch rather than continuous.

The design's own description hints at the fallback path: "if tweets are not found then look into the backend index server." That is a cascading lookup — try fast, fall back to slow — which is the right default given that most queries never reach the second tier.

The two search products are different systems wearing one name

One search type returns the result of the last seven days, which all registered users usually use.

The second query type searches the full historical corpus, including archived content. Results may include content dating back to the earliest tweets. This type is typically used for research and analytical workloads.

Notice the design calls these different products, not different queries:

That is a clean commercial resolution of a technical problem. Deep historical search is genuinely expensive, so rather than making it fast for everyone, it becomes a separate product with different latency expectations and a different price.

When one workload is orders of magnitude more expensive than another, making it a separate product is a legitimate architectural decision. It aligns cost with the users who create it, and it lets you size the fast path for the common case without over-provisioning for the rare one.

This also explains Lesson 4's odd 3,200-result cap. If the interactive index is a bounded window, deep pagination eventually walks off it — so the cap is where the fast path ends, not an arbitrary product limit.

A trillion records in under 100 ms — where the budget goes

Worth decomposing, because the number sounds impossible and is not.

An inverted index does not scan a trillion records. It maps each term to a posting list of documents containing it, so a query for "earthquake" touches one list, not the corpus. The trillion is the corpus size, not the work per query.

The 100 ms budget then covers roughly:

parse and analyze the query           ~1 ms
fetch posting lists (RAM)             ~1-10 ms
intersect / rank                      ~10-30 ms
fan out across index shards           bounded by SLOWEST shard
hydrate tweet bodies from cache       ~10 ms

The fan-out line is the one that dominates in practice. The index is sharded across many servers, every query goes to all of them, and the response waits for the slowest — the tail-latency problem from distributed search, which appeared again in the Maps and Yelp fan-outs.

Which is why the RAM residency matters so much. A disk seek on any one shard would blow the budget for the entire query, so the fast path cannot afford to touch disk at all. That is the real argument for keeping the hot week in memory — not average latency, but the tail.

Partition by document, not by term. Term partitioning looks efficient — one shard per word — until a two-word query has to intersect posting lists that live on different machines, which means shipping large lists over the network. Document partitioning makes every shard able to answer independently, and the cost is that every query touches every shard.

Next: caching, where the objects are small enough that bookkeeping costs money.