Request Types: CPU-, Memory-, and IO-Bound

In one line: "requests per second" implies all requests are alike. They are not, and the difference between the cheapest and most expensive class is roughly a hundredfold — which means guessing wrong about your workload's class invalidates the whole estimate.

The three classes

Using the latency table from the previous lesson as the reference:

Class	Limited by	Example	Time	Relative to CPU-bound
CPU-bound	Processing speed	Compressing 1 KB of data	3 us	1x
Memory-bound	The memory subsystem	Reading 1 MB sequentially from RAM	9 us	~3x slower
IO-bound	The IO subsystem (disk or network)	Reading 1 MB sequentially from SSD	200 us	~66x slower

To simplify calculations, these differences are commonly approximated as clean orders of magnitude:

CPU-bound task    = X
Memory-bound task = 10X
IO-bound task     = 100X

The measured ratios (1x, 3x, 66x) are close enough to 1/10/100 for estimation purposes, and the round numbers are far easier to carry through a multi-step calculation.

Identifying which one you have

Workload	Dominant class	Tell
Video transcoding, encryption, compression	CPU-bound	Cores are pinned; disk and network idle
In-memory analytics, large sorts, joins	Memory-bound	High memory bandwidth use; CPU waits on RAM
Serving files, database queries, calling other services	IO-bound	Low CPU utilization while latency is high
Typical web API endpoint	IO-bound	Most time spent waiting on a database or a downstream call

When the workload class shifts

Suppose a service transitions from mostly CPU-bound to mostly IO-bound — a common trajectory as a product grows, since features that once computed in memory start reading from a database or calling other services. What changes in your planning?

Planning dimension	CPU-bound service	IO-bound service
Per-request cost	~3 us — roughly 1,000 requests/sec/core	~200 us or worse — orders of magnitude fewer
What to buy	More cores, faster clocks	Faster storage, more IOPS, more bandwidth, caching
Concurrency model	Threads ~ cores; more just adds context switching	Concurrency far exceeds cores — threads are blocked, not busy
Server count	Scales with core count	Driven by wait time, not compute — Little's Law governs
Biggest lever	Better algorithms, SIMD, compression choice	Caching, batching, denormalization, fewer round trips
What saturates first	CPU utilization	Connection pool, IOPS ceiling, or NIC

The headline consequence: your per-server request estimate drops by up to two orders of magnitude, so a server count derived from CPU-bound assumptions becomes badly optimistic. But the fix is usually not more servers — it is removing the IO, via caching, batching, and cutting round trips. Recall Little's Law from the previous chapter: with IO-bound work, concurrency must rise to cover the wait, or throughput collapses regardless of how many cores are idle.

Key takeaway

Classify the workload before you size it. A CPU-bound estimate applied to an IO-bound service will be optimistic by up to 100x — and will point you at the wrong hardware to fix it.

Interview signal by level

Level	What a strong answer sounds like
L4	Treats all requests as equally expensive.
L5	Distinguishes them: "this endpoint is IO-bound — it's waiting on the database, not computing."
Staff+	Sizes from the class: "IO-bound at roughly 100x a CPU-bound request, so my per-server RPS drops by two orders of magnitude. That means caching and batching before capacity — and concurrency has to exceed core count because those threads are blocked, not busy."

Next: deriving a server's request capacity from first principles.