A Latency Budget Set by Human Typing
In one line: most latency requirements are chosen. This one is measured from human motor behaviour, and it is tight enough that it dictates the data structure, the update strategy, and where the work is allowed to happen.
The requirement that defines the system
Low latency: the system must provide suggestions in real time, with latency under 200 ms. Since the average time between keystrokes is 160 ms, the system must respond quickly to remain useful and prevent stale suggestions.
A budget derived from physiology, not from product preference
Read the two numbers together, because their relationship is the whole design:
Average time between keystrokes: 160 ms
Latency budget: 200 ms
-------
Slack: 40 ms
If a suggestion takes longer than the gap between keystrokes, it arrives after the user has already typed the next character — which makes it a suggestion for a prefix that no longer exists. The response is not merely late; it is answering the wrong question.
That is a different kind of deadline from the ones in earlier chapters. Google Maps wanted routes in 2–3 seconds because users get impatient. Here, exceeding the budget makes the output incorrect, because the input has changed underneath it.
Three consequences follow, and they determine everything after:
No database read on the request path. "reading from a database is much slower than reading from RAM. Therefore, we must store the index in memory." A disk seek alone can consume the entire budget.
No computation on the request path. Ranking, aggregation, and frequency counting all move offline — which is Lesson 7's assembler.
No traversal if it can be avoided. Lesson 5's precomputed top-k inside each node exists to turn a tree walk into a lookup.
When the deadline is set by the rate at which the input changes, a late answer is a wrong answer — and that forces every expensive operation off the critical path.
The latency budget is not a target someone chose. It is set by how fast humans type, and it is what forces every later decision.
What the system does
The system suggests the top N frequent and relevant terms based on the user's input.
One functional requirement. That is unusually spare, and it is honest — the product is a single operation performed extremely well.
The workload is unusual: many requests, one tiny answer, enormous read skew
Worth characterizing before designing, because the shape is distinctive:
| Property | This system |
|---|---|
| Request rate | Very high — one per keystroke, not per search |
| Payload in | Tiny — a prefix, a few bytes |
| Payload out | Tiny — ten short strings |
| Computation per request | Should be zero — a lookup |
| Read/write ratio | Overwhelmingly read |
| Data volatility | Low — popular queries change slowly |
Two of those rows do most of the work.
"One request per keystroke" multiplies the search volume by the query length. Lesson 3 shows 3.5 billion searches becoming 52.5 billion characters, so the typeahead system sees roughly fifteen times the request rate of the search engine behind it.
"Data volatility is low" is what makes the whole design possible. The design states it plainly: "top suggestions do not change frequently enough to require immediate updates." If popular queries churned by the second, you could not precompute anything, and the 200 ms budget would be unmeetable.
A tight latency budget is affordable only when the data is stable enough to precompute. That pairing — strict deadline, slow-changing data — is what licenses the offline assembler in Lesson 7.
What is notable about this chapter
It is the first source in the module to catch its own estimation error
Lesson 3 covers this in full, and it deserves flagging early because it is genuinely unusual.
The chapter computes a server count of 821,000, and then writes:
Note: The initial estimate of 52.5 billion requests per second is unrealistic.
No other chapter in this module notices. Yelp, Newsfeed, Instagram, TinyURL, and the web crawler all produced figures off by three to five orders of magnitude and presented them without comment.
The catch is that the correction lands on 164,000 servers, which is still wrong by a factor of 17,000 — and the chapter's own interactive table contains the right answer, 9.
Prose, first attempt: 821,000 servers <- flagged as unrealistic Prose, "revised": 164,000 servers <- still 17,280x too high Interactive table: 9 servers <- correct
Noticing that a number is implausible is most of the skill; the rest is finding which input was wrong. This chapter does the first half and not the second, which is a more interesting failure than not noticing at all.
Where the difficulty actually lives
Given that the request rate is high and the answer is tiny, the design problems are:
| Problem | Where |
|---|---|
| A structure that answers prefix queries in memory | The trie — Lesson 5 |
| Ranking without computing at request time | Precomputed top-k in nodes — Lesson 5 |
| Splitting it across machines | Prefix-range partitioning and its skew — Lesson 6 |
| Updating without blocking reads | Offline assembler and replica swap — Lessons 6 and 7 |
| Not sending a request at all | Client-side debouncing — Lesson 8 |
The last one is worth noticing now. Because the request rate is driven by keystrokes, the highest-leverage optimization is suppressing requests before they leave the browser — the same "avoid the work entirely" instinct the web crawler chapter was built on.
Key takeaway
The latency budget is derived from human motor behaviour — 200 ms against a 160 ms inter-keystroke interval — and exceeding it does not merely make the answer late, it makes it answer a prefix that no longer exists. That forces every expensive operation off the request path: no database read, no computation, ideally no traversal. The workload is one request per keystroke, so the system sees roughly fifteen times the search engine's rate — but data volatility is low, and a tight deadline is affordable only when the data is stable enough to precompute. And this is the first source in the module to flag its own estimate as unrealistic, though it corrects to a figure still 17,000× too high while its own table holds the right one.
Next: the requirements and the estimation.