How Many Requests Can One Server Handle?

Why this matters: every server-count estimate divides total load by per-server capacity. That denominator has to come from somewhere, and deriving it beats guessing — because you can then update it honestly when an assumption changes.

Key takeaway

Estimate a server's throughput by computing the CPU time required per request, then dividing one second by it and multiplying by core count. With reasonable assumptions this yields 1,000 requests/sec per core, or 64,000 RPS for a 64-core server.

The setup

A real request touches many nodes. For estimation we accumulate all that work into a single figure and ask how long one processor spends on one request.

The governing equation:

CPU time per program = Instructions per program * CPI * CPU time per clock cycle

Where CPI is clock cycles per instruction. Dimensional analysis confirms the result is in seconds:

Term	Meaning	Unit
Instructions per program	Instruction count for one request	Unitless
CPI	Cycles per instruction	Unitless
CPU time per clock cycle	Duration of one clock cycle	Seconds

Multiplying gives CPU time per request, in seconds.

The assumptions

Three, all stated explicitly so they can be challenged:

CPI                        = 1
Processor clock rate       = 3.5 GHz  (3.5 billion cycles per second)
Instructions per request   = 3.5 million

Working it through

First, the time for a single clock cycle at 3.5 GHz:

Clock cycles per second = 3.5 * 10^9

CPU time per clock cycle = 1 / (3.5 * 10^9)

Then substitute into the equation:

CPU time per program = (3.5 * 10^6) * 1 * 1 / (3.5 * 10^9)
                     = 0.001 second

The 3.5 cancels cleanly — which is why those particular assumptions were chosen. One request costs one millisecond of CPU time.

Now invert to get throughput per core, then scale by cores:

Requests one CPU core executes in 1 second = 1 / 10^-3 = 1,000 requests

Requests a 64-core server executes in 1 second = 64 * 1,000 = 64,000 requests

64,000 RPS is the number carried through the rest of this chapter.

What this deliberately ignores

Changing the assumptions changes the estimate. That is a feature — but only if you know which assumptions are load-bearing:

Assumption	Stated value	Reality	Effect if wrong
CPI = 1	One cycle per instruction	Cache misses and branch mispredicts push it higher	CPI of 2 halves the estimate
3.5M instructions/request	A moderate request	Varies enormously by endpoint	Directly proportional — 10x instructions, 10x fewer requests
Perfect core scaling	64 cores = 64x one core	Contention, locks, and memory bandwidth intervene	Real scaling is sublinear
Purely CPU-bound	No waiting	Most services wait on IO	Can be 100x optimistic

Key takeaway

Derive your per-server capacity, state the assumptions that produced it, and label it as a ceiling. A number you can reconstruct is one you can defend when the interviewer pushes on it — which they will.

Interview signal by level

Level	What a strong answer sounds like
L4	Asserts a capacity figure with no derivation.
L5	Derives it: "about 1 ms of CPU per request, so 1,000 per core, 64,000 on a 64-core box."
Staff+	Derives and bounds it: "that's a CPU-only ceiling assuming CPI of 1 and perfect core scaling. Our workload is IO-bound, so I'd plan against something one to two orders of magnitude lower and treat 64,000 as the number to aim for after optimization."

Next: applying this to a real service.