Request Types: CPU-, Memory-, and IO-Bound

Why this matters: "requests per second" implies all requests are alike. They are not, and the difference between the cheapest and most expensive class is roughly a hundredfold — which means guessing wrong about your workload's class invalidates the whole estimate.

Key takeaway

Real workloads fall into three categories — CPU-bound, memory-bound, and IO-bound — separated by roughly an order of magnitude each. Knowing which one dominates tells you what resource to buy and what to optimize.

The three classes

Using the latency table from the previous lesson as the reference:

Class	Limited by	Example	Time	Relative to CPU-bound
CPU-bound	Processing speed	Compressing 1 KB of data	3 us	1x
Memory-bound	The memory subsystem	Reading 1 MB sequentially from RAM	9 us	~3x slower
IO-bound	The IO subsystem (disk or network)	Reading 1 MB sequentially from SSD	200 us	~66x slower

To simplify calculations, these differences are commonly approximated as clean orders of magnitude:

CPU-bound task    = X
Memory-bound task = 10X
IO-bound task     = 100X

The measured ratios (1x, 3x, 66x) are close enough to 1/10/100 for estimation purposes, and the round numbers are far easier to carry through a multi-step calculation.

Identifying which one you have

Workload	Dominant class	Tell
Video transcoding, encryption, compression	CPU-bound	Cores are pinned; disk and network idle
In-memory analytics, large sorts, joins	Memory-bound	High memory bandwidth use; CPU waits on RAM
Serving files, database queries, calling other services	IO-bound	Low CPU utilization while latency is high
Typical web API endpoint	IO-bound	Most time spent waiting on a database or a downstream call

When the workload class shifts

Suppose a service transitions from mostly CPU-bound to mostly IO-bound — a common trajectory as a product grows, since features that once computed in memory start reading from a database or calling other services. What changes in your planning?

Planning dimension	CPU-bound service	IO-bound service
Per-request cost	~3 us — roughly 1,000 requests/sec/core	~200 us or worse — orders of magnitude fewer
What to buy	More cores, faster clocks	Faster storage, more IOPS, more bandwidth, caching
Concurrency model	Threads ~ cores; more just adds context switching	Concurrency far exceeds cores — threads are blocked, not busy
Server count	Scales with core count	Driven by wait time, not compute — Little's Law governs
Biggest lever	Better algorithms, SIMD, compression choice	Caching, batching, denormalization, fewer round trips
What saturates first	CPU utilization	Connection pool, IOPS ceiling, or NIC

The headline consequence: your per-server request estimate drops by up to two orders of magnitude, so a server count derived from CPU-bound assumptions becomes badly optimistic. But the fix is usually not more servers — it is removing the IO, via caching, batching, and cutting round trips. Recall Little's Law from the previous chapter: with IO-bound work, concurrency must rise to cover the wait, or throughput collapses regardless of how many cores are idle.

Key takeaway

Classify the workload before you size it. A CPU-bound estimate applied to an IO-bound service will be optimistic by up to 100x — and will point you at the wrong hardware to fix it.

Interview signal by level

Level	What a strong answer sounds like
L4	Treats all requests as equally expensive.
L5	Distinguishes them: "this endpoint is IO-bound — it's waiting on the database, not computing."
Staff+	Sizes from the class: "IO-bound at roughly 100x a CPU-bound request, so my per-server RPS drops by two orders of magnitude. That means caching and batching before capacity — and concurrency has to exceed core count because those threads are blocked, not busy."

Next: deriving a server's request capacity from first principles.