Initial Design: Servers, Stores, and Caches

In one line: what sits between the web and application tiers is a work queue, not a load balancer — and the two caches are split by operation, not by data.

The baseline architecture

Four component groups: web and application servers joined by a custom routing layer, three data stores chosen by data shape, two distributed caches, and separate compute hosts for ML.

The routing layer is a work queue

Web servers accept HTTP requests and run manager processes that delegate work to worker processes on application servers through a routing layer. That layer manages task queues: managers enqueue, workers dequeue.

An interviewer will ask why this isn't just a load balancer. The answer is in one phrase — one manager generates multiple worker tasks.

A load balancer routes one request to one backend. It has no concept of fanning a single request into several parallel units of work. What sits here is a work queue: the manager decomposes a page into tasks, any free worker takes one, and the results are assembled.

Using a load balancer would force one request onto one worker and serialize what should be parallel.

Two payoffs. Perceived latency drops, because the user sees page structure before all the data arrives. And the slow components run concurrently, so total time is the slowest one rather than the sum.

Three stores, three different reasons

The MySQL and blob-storage choices are familiar. HBase's justification carries something less obvious.

"Avoids expensive recomputation of features" means HBase is a materialized cache of ML feature values. Extracting features from raw content — text analysis, engagement history, topic signals — is costly, so you compute once and store the result.

That reframes it from "a NoSQL store for high-throughput data" to "where computed ML inputs live so the model doesn't recompute them per request."

Hold onto this, because it sets up the chapter's sharpest lesson: the store was chosen for write throughput and will turn out to sit on a latency-critical read path. That is a very common way for a storage choice to age.

Two caches, split by operation

This looks like duplicated infrastructure and is a genuine capability difference.

Memcached is a cache of values. You read a value, you write a value. To increment a counter you would read it, add one, and write it back — a read-modify-write race that loses updates under concurrency.

Redis supports atomic in-store operations. INCR increments server-side in one indivisible step, so concurrent increments cannot lose each other.

For view counters — high-frequency, concurrent, per-answer increments — that difference is decisive. Memcached would silently undercount.

So the split is by operation, not by data: Memcached caches things you read, Redis holds things you mutate.

Note the limit of the fix. Redis solves the concurrency correctness; it does not solve a single scorching key. For a genuinely hot counter you want sharded counters, and in the final design view counting moves to Kafka for asynchronous processing anyway.

CDNs serve the static media on top of all this.

ML on its own hosts

Compute servers handle recommendations and ranking, using ML in both online and offline modes, and they need substantial RAM and CPU.

Putting that on separate hosts rather than on application servers is the same reasoning as separating indexing from search. The two workloads have incompatible profiles: application servers handle many short latency-sensitive requests, while inference and feature extraction are compute- and memory-hungry batch-ish work. Colocated, they contend — and worse, they scale on different signals, request volume versus model complexity and corpus size.

The payoff is reuse. One feature pipeline serves ranking, feed, ad targeting, and moderation — four products from one system, which is only possible because it is a separate service rather than logic embedded in the request path.

Interview signal by level

Next: the workflows those components carry out.