High-Level Design and the API
In one line: the offline/online split is the architecture. Everything expensive happens on one side of the line, and everything latency-critical on the other.
The architecture
| Component | Role |
|---|---|
| Crawler | Collects content from sources, extracts text and metadata to enable intelligent search based on titles, descriptions, and video content, formats it as JSON, and stores it in a distributed storage system |
| Indexer | Fetches documents and builds indexes using MapReduce on a distributed cluster. The resulting index table is stored in a distributed storage system |
| Distributed storage | Holds the raw documents and the constructed index |
| User | Submits a search string containing one or more words |
| Searcher | Parses the query, maps terms to the index, and returns ranked results. Also handles spell correction and relevance ranking |
Distributed storage is the seam — the two phases never talk to each other directly
Trace the arrows and notice something: the crawler never calls the indexer, and the indexer never calls the searcher. Everything goes through distributed storage.
That is the same decoupling pub-sub argued for, applied to a pipeline. Three consequences:
Independent failure. The indexer can be down for an hour and search keeps serving from the last index. The crawler can fall behind and indexing continues on what it has.
Independent scaling. Lesson 12 lists this as how scalability is achieved — the searcher scales with query volume, the indexer with corpus volume, and neither constrains the other.
Independent deployment. You can rewrite the indexer without touching the search path, because the contract between them is a file format, not an API.
Lesson 5 noticed that even the centralized design communicated through a binary index file. This is that same seam, widened.
The crawler extracts more than text, and that shapes what search can do
"Extracts text and metadata to enable intelligent search based on titles, descriptions, and video content" is doing real work.
You cannot index a video — you index text about a video. Titles, descriptions, channel names, and transcripts are what make video searchable at all, and Lesson 2's storage model is exactly that: a 200 KB JSON document per video.
Two things follow. Search quality is bounded by extraction quality — a video with a poor title and no transcript is effectively invisible regardless of how good the index is. And the crawler is where domain knowledge lives: crawling YouTube, an e-commerce catalogue, or a codebase means extracting completely different fields, while the indexer and searcher stay generic.
That is a good separation to name: the crawler is domain-specific; everything downstream operates on documents.
The API
The API is straightforward since users send string requests.
search(query)
| Parameter | Description |
|---|---|
query | The textual query entered by the user in the search bar, based on which the results are found |
One parameter, and that simplicity is the point
The entire user-facing surface of a system spanning crawlers, MapReduce clusters, and distributed storage is one function with one string argument.
That is the abstraction working. Everything in this chapter — partitioning, replication, merging, ranking — exists below this line and is invisible above it. The user does not choose a partition, does not know how many nodes answered, and does not see the merge.
Compare object storage's seven-call API, where users manage containers and paths explicitly. Here the system exposes no structure at all, which is why it can restructure everything underneath without breaking a single client.
In practice this grows parameters — pagination, filters, language, safe-search — but the shape holds: the query is a string, and the system owns everything else.
What the one-parameter API hides is where the difficulty is
Because the API is a string in and results out, the hard requirements are all implicit:
- How many results? Lesson 2's response size answers it: 80 suggestions. Not in the signature.
- In what order? Relevance ranking — the harder half from Lesson 4, and entirely invisible here.
- What if the query is misspelled? Spell correction, listed as the searcher's job.
- How fresh? Whatever the last completed index run produced — the staleness the offline phase buys.
An interviewer asking "design search" and hearing only search(query) will push on exactly these. Volunteering them — top-K, ranked, typo-tolerant, and slightly stale — turns a thin API into a demonstration that you know what it is concealing.
Key takeaway
Two phases: offline crawling and indexing, online serving. They communicate only through distributed storage, which gives independent failure, scaling, and deployment. The API is one string parameter — every decision in this chapter lives below that line, which is what makes the system free to restructure.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "Crawler collects, indexer indexes, searcher searches, and the API is search(query)." |
| L5 | Names the phase split: "crawling and indexing are offline and go through distributed storage; only the searcher is online, so it never waits on the indexing pipeline." |
| Staff+ | Reads the seam and the hidden requirements: "the components never call each other — the contract between them is a file in distributed storage, which is what lets us fail, scale, and deploy them independently. And the one-parameter API hides the actual requirements: top-K, ranked by relevance, typo-tolerant, and deliberately slightly stale. I'd state those explicitly, because they're where the design work is." |
Next: how to split the data.