MyRocks and Kafka

In one line: the MyRocks change comes with a real published number — p99 from 80 ms to 4 ms — which makes it one of the few storage swaps in this course with measured justification rather than an argument.

MyRocks

80 ms to 4 ms at p99 is a twentyfold improvement, and it removes the store from the latency budget

Lesson 6 diagnosed the problem as two slow things in series: read features from HBase, then run ML inference. At an 80 ms p99 for the read alone, the store was a substantial share of the total.

At 4 ms, it essentially disappears — inference now dominates, which is where the time should be spent.

Two things worth extracting:

The metric is p99, not average. The Non-Functional Characteristics chapter's point applies: users experience the tail, and at Quora's request volume p99 occurs constantly. A store with a good average and a bad tail is a bad store for a request path.

Replacing beats caching. Lesson 4 noted the search index was cached "for low latency" — mitigating HBase's tail rather than fixing it. Caching around a latency problem helps on hits and leaves the tail intact on misses. Changing the store fixes it everywhere.

The general instinct: when you find yourself adding a cache to hide a store's latency, ask whether the store is wrong for the access pattern. Sometimes the cache is right; sometimes it is a workaround for a choice that should be revisited.

The second reason is operational, and it is the one that made adoption feasible

"Existing tools facilitate data transfer between MyRocks and MySQL" sounds minor and is not.

Lesson 3 established the pipeline: the ML engine extracts features from MySQL data, and those features are stored for reuse. So there is a constant flow from MySQL into the key-value store.

With HBase that flow crosses an ecosystem boundary — different storage engine, different tooling, custom pipelines. MyRocks is a MySQL storage engine, so the same tools, the same replication mechanisms, and the same operational knowledge apply.

That is often what decides real migrations. A store that is 20x faster but requires an entirely new operational stack may lose to one that is fast and fits what the team already runs. Operational fit is a real selection criterion, not a soft one — and it is worth naming in an interview, because candidates usually compare only on performance.

Kafka

To reduce load on service hosts, we offload low-priority tasks to Kafka. Kafka distributes jobs like view counting, notifications, and analytics to various queues. These are processed via cron jobs.

This fixes both halves of Lesson 6's in-memory queue problem

Lesson 6 identified two failures with in-memory priority queues, and Kafka addresses both:

Durability. Queues in application-server memory lose everything on a crash, requiring "manual reprocessing." Kafka is a durable log — the Pub-Sub chapter's append-only segments with retention — so a consumer crash costs nothing; work resumes from the stored offset.

Decoupling. Background tasks were competing for memory with request serving on the same hosts. Moving them to Kafka takes them off the service hosts entirely, so "application servers should not be saturated with low-priority background tasks" becomes structurally true rather than a hope.

Note the specific tasks named — view counting, notifications, analytics — which are exactly the examples Lesson 6 called out as having no business on a page-serving machine. And it composes: Lesson 4 put notifications in a low-priority queue, and Kafka is now what that queue is.

This fixes both halves of the in-memory queue problem at once: the log is durable, so a crash no longer loses work, and it is external, so background volume stops pressuring the memory of machines serving page loads.

Why view counting through Kafka rather than synchronously

"In our detailed design, we have employed Kafka to handle our view counter of answers. Why?"

For popular topics, a large number of users may access the same answer concurrently. This can saturate application and database resources. Strictly real-time updates of the view counter are not required. Instead, view events can be published to Kafka and processed asynchronously, decoupling write spikes from the primary request path.

Quora handles these tasks in two minutes or less. Nonetheless, we can use sharded counters as an effective solution to the view counter problem.

Two distinct mechanisms, and it is worth separating them:

Kafka decouples timing. A burst of views becomes a burst of log appends — cheap and sequential — rather than a burst of counter updates. The spike is absorbed by the log and drained at a steady rate. That is message-queue design's buffering argument, and the two-minute lag is the price.

Sharded counters decouple contention. Kafka smooths the arrival rate but does not fix the fact that a popular answer's counter is a single hot key — the sharded-counters pattern's heavy-hitters problem. If the aggregate rate for one answer exceeds what one counter can absorb, buffering only delays the collapse.

So both apply: Kafka for the spike, sharded counters for the hot key. They solve different halves, and a genuinely viral answer needs both.

'Processed via cron jobs' is a simple choice with a real limitation

Draining Kafka with cron jobs rather than continuous consumers is worth noticing.

It fits the workload — these are delay-tolerant tasks and the two-minute target is easily met by frequent cron runs. It is also operationally simple: no long-running consumer processes to supervise.

The limitation is granularity. A cron job runs on a fixed schedule regardless of queue depth, so it cannot react to a surge by consuming faster. Under an unusual spike, lag grows until the next scheduled run, and each run has more to do.

A continuous consumer group would scale with demand and keep lag bounded. For Quora's stated two-minute target this is fine; it is worth knowing as the thing that would change if the latency requirement tightened.

Interview signal by level

Next: the technology choices around them.