Performance: Latency, Throughput, and the Tail

Why this matters: availability decides whether users can reach you. Performance decides whether they stay. And the way most engineers measure it — averages — systematically hides the experience of the customers most likely to leave.

Key takeaway

Performance measures a system's ability to respond to requests and process data efficiently. It has two distinct axes — latency (how long one request takes) and throughput (how many requests per second) — and optimizing one often costs the other.

Latency and throughput are not the same

Metric	Question	Units	Improved by
Latency	How long does one request take?	Milliseconds	Caching, fewer hops, faster algorithms, colocation
Throughput	How many requests per second?	Requests/sec	Parallelism, batching, more capacity

They routinely conflict. Batching raises throughput by amortizing per-request overhead — and raises latency, because a request now waits for the batch to fill. A system tuned purely for throughput can feel terrible to use.

Why averages lie

Average latency is the most misleading number in system design. Consider 100 requests: 99 take 10 ms and one takes 2,000 ms. The average is about 30 ms, which sounds excellent — and 1% of your users waited two seconds.

Use percentiles:

Percentile	Meaning	Who it represents
p50 (median)	Half of requests are faster	The typical experience
p95	1 in 20 requests is slower	Noticeably annoyed users
p99	1 in 100 is slower	Where you set SLOs
p99.9	1 in 1,000 is slower	Often your largest, most valuable customers

Lever 1: Caching

Caching stores frequently accessed data, reducing repeated computation and user-perceived latency.

The celebrity fan-out problem

A concrete case worth knowing, because interviewers use it constantly. Consider an X (formerly Twitter)-like system with a service that generates each user's timeline.

When a celebrity posts, does the service generate a timeline for every follower? With millions of followers, that write amplifies into millions of updates and performance collapses.

The resolution is to split users by behavior:

Inactive users — generate the timeline on demand, when they actually open the app. Most never will, so the work is never done.
Active users — maintain a feed cache that is prepopulated. When they request their feed, the service returns it immediately.

This is the general shape of the fix: do expensive work only for the users who will actually observe it, and precompute only what has a reader. Caching at multiple layers further ensures decoupling and low latency.

Lever 2: Algorithm and data-structure selection

Efficient algorithms minimize processing time, and the right choice depends on the access pattern, not on which structure is theoretically fastest.

Consider a ride-hailing system where every driver reports position every four seconds. You need a structure that handles frequent spatial updates efficiently.

A quadtree is a strong candidate for spatial indexing — it answers "who is near this point?" efficiently. But updating a quadtree every four seconds for every driver introduces real computational overhead, and that overhead becomes latency. The structure that is excellent for reads is expensive for this write rate.

So the honest answer is to evaluate whether a quadtree is optimal here at all, or whether a hybrid better balances performance and scalability — for instance, a coarser grid for high-frequency position updates with a finer index rebuilt periodically.

Lever 3: Load distribution

Distributing traffic evenly across servers prevents bottlenecks. For an e-commerce site handling millions of concurrent requests, load balancers ensure no single server is overwhelmed — one saturated server means queueing, and queueing is latency.

This also connects performance back to protocol choice. For a messaging service asked to deliver messages with low latency, an efficient two-way protocol like WebSocket avoids repeated connection setup and lets the server push rather than making clients poll.

Putting the levers together

Symptom	Likely cause	Lever
Everything is uniformly slow	Under-provisioned or an inefficient hot path	Profile first; then capacity
p50 fine, p99 terrible	Tail — GC pauses, cold caches, slow shard, fan-out	Hedged requests, cache warming, find the slow shard
Slow only under load	Queueing — concurrency limit reached	Little's Law: raise concurrency or cut latency
Slow for some users only	Data skew — heavy accounts, hot partition	Split the hot key; paginate; bound per-user work
Fast locally, slow in production	Network round trips and real data volumes	Batch calls, colocate, add indexes