Limitations of the Initial Design
In one line: each of these four becomes a fix in the next four lessons, and being able to find them yourself is what separates evaluating a design from reciting one.
They all have the same shape
Read them together and the diagnostic is worth keeping: when a system underperforms, look for two things sharing a resource or a critical path that should not.
1. The extra network hop
Payloads are exchanged between web and application servers to process a request, adding network I/O latency. The tier separation enables parallel processing through managers and workers, but the extra hop increases end-to-end response time.
This is a genuine tension, not an obvious mistake.
Lesson 3 justified splitting the tiers: managers fan a page into parallel worker tasks, which is real and valuable. But every task crossing that boundary is a network round trip, plus routing coordination. If one page produces several worker tasks, you pay that cost several times per request.
So the question is whether parallelism gained exceeds latency added, and Lesson 7's answer is that it does not. The fix collapses both tiers onto one host, keeping the manager-worker decomposition but making it inter-process rather than inter-machine.
Separating tiers is not free, and the cost is paid per interaction. Service boundaries earn their keep when tiers scale differently or fail independently; they do not when the only benefit is organizational and the traffic between them is chatty.
Note what survives the fix — the work-queue model is kept. What is removed is the network between producer and consumer.
2. In-memory queues are a single point of failure
The application server enqueues tasks into in-memory priority queues. If a queue crashes or its process terminates, queued tasks are dropped and need manual reprocessing. Replicating them costs memory, and as feature count grows, background task volume grows with it.
Two failures, and the second is the structural one.
Durability. A process crash takes every queued task with it. A queue that loses acknowledged work is not a queue you can build on.
Coupling. Background tasks compete for the application server's memory with the request serving that is its actual job. As features accumulate, background volume grows — on the same machines handling user requests.
That second point is the colocation problem again: two workloads with different profiles on the same hosts, unable to scale independently. Here it is worse, because the background work is delay-tolerant and is nonetheless allowed to pressure latency-critical work.
The named examples are exactly right — view counters and analytics events have no business consuming memory on a machine serving page loads. Lesson 9 moves them to Kafka: durable, external, independently scalable.
3. Hot MySQL tables
As the system supports more features, a small subset of MySQL tables receives a disproportionate share of queries, driving high QPS on specific instances and increasing contention and latency.
Load is not evenly distributed across tables. A handful — questions, answers, votes — absorb most queries while dozens sit nearly idle.
That is a heavy-hitters shape at table granularity rather than row granularity. And it means adding MySQL capacity does not help: a bigger server still serializes on the same tables.
The fix has to separate the hot tables from each other, which is what Lesson 8's vertical sharding does — split by table rather than by row.
The design also defines no disaster recovery strategy, which Lesson 12 addresses.
4. HBase tail latency
HBase supports high write throughput, but its p99 is relatively high. Several features depend on an ML inference pipeline that adds its own latency. Combined, they raise end-to-end response time.
Neither component alone is disqualifying. Together they are, because serial latencies add while the user experiences the total.
Two things worth extracting.
Tail latency, not average, is what matters. HBase's average may be fine; p99 is what users notice, and at Quora's request volume p99 happens constantly.
Choose the store for the access pattern that will actually run against it. HBase was picked in Lesson 3 for write throughput and for holding pre-computed features. It turned out to sit on a latency-critical read path, which is not what it was selected for.
Lesson 9 replaces it with MyRocks, taking p99 from 80 ms to 4 ms — removing the store from the latency budget almost entirely.
Key takeaway
Four limitations, all coupling problems. The tier split buys parallelism and pays a network hop per task. In-memory queues lose work on crash and let background load pressure request-serving memory. A few MySQL tables take most of the traffic, so more capacity does not help — this is skew, not volume, so the fix is vertical rather than horizontal. And HBase p99 sits in series with inference, where serial latencies add and the tail is what users feel.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "We could add more servers and more caching if it gets slow." |
| L5 | Finds specific problems: "the in-memory queues lose work if a process dies, and a few hot tables will bottleneck MySQL before the rest of it does." |
| Staff+ | Names the common shape and the right fix for each: "all four are coupling — a hop paid per task, background work sharing request-serving memory, hot tables sharing an instance, and a slow store in series with slow inference. And the MySQL one is skew rather than volume, so it's vertical sharding, not horizontal — a bigger box still serializes on the same tables. I'd also flag that caching the hot index entries is mitigating HBase's p99 rather than fixing it; the fix is a different store." |
Next: collapsing the tiers.