Scalability Techniques and Their Trade-Offs

Why this matters: knowing the techniques is table stakes. Knowing the order to apply them, and what each one costs, is what makes an answer sound like it came from someone who has done it.

Key takeaway

Every scalability technique removes one specific bottleneck and adds one specific cost. Applied in the right order — cheapest and least invasive first — a handful of them carry a system from one server to global scale.

The scaling ladder

Reach for these roughly in this sequence. Each rung buys time and adds complexity; climbing too fast means paying for coordination problems you don't have yet.

Load balancing

Load balancing distributes user traffic evenly across servers, preventing overload and failure on any single node.

It is the enabler for horizontal scaling: without it, adding servers gives you nothing because nothing routes traffic to them. It also provides the health-checking and outlier ejection that turn redundancy into actual fault tolerance.

Cost: the load balancer itself becomes a critical component, which means it needs its own redundancy. Precondition: the app tier must be stateless, or you need sticky sessions and lose much of the benefit.

Caching and CDNs

Caching stores frequently accessed data in fast temporary storage, reducing database load. CDNs distribute static content — videos, images — via servers geographically closer to users.

This is usually the highest-leverage move available. Most workloads are read-heavy and follow a power law — a small fraction of items serve most of the traffic — so a modest cache absorbs the large majority of reads. A CDN goes further and removes the request from your infrastructure entirely.

Cost: cache invalidation, and stale reads. You have just chosen eventual consistency for cached data, so everything from Chapter 1 applies — including read-your-writes for the user who just changed something.

Replication and sharding

Two different techniques that are often confused, because both involve multiple databases:

Technique	What it does	Scales	Does not help with
Replication	Duplicates the same data across servers	Reads, and fault tolerance	Write throughput — every replica takes every write
Sharding	Partitions different data across servers	Writes, and total data volume	Fault tolerance — each shard is still a single copy

You generally need both: shard for write and storage capacity, replicate each shard for durability and read scale.

Cost of replication: replica lag, and the consistency decisions that come with it. Cost of sharding: choosing a partition key you cannot easily change, losing cross-shard transactions and joins, and the risk of a hot shard if the key distributes unevenly.

Asynchronous workers

Efficient resource utilization means using queuing mechanisms for incoming requests and worker servers for background tasks.

Worker servers handle time-consuming tasks that don't require immediate user feedback — processing images, running data analysis jobs — so the platform's core functions stay fast. The user's request returns as soon as the work is accepted, not when it is finished.

This is the load-absorption mechanism that autoscaling cannot provide: the queue takes the spike instantly while workers drain at their own pace.

Cost: eventual consistency, plus the operational machinery from Chapter 1 — duplicate handling, dead-letter queues, and consumer lag monitoring.

Service decomposition

Microservices architecture decomposes applications into independent services, each scaling independently based on its own demand.

The real win is independent scaling and independent deployment: video transcoding needs enormous CPU while the user profile service needs almost none, and in a monolith you scale both together. Teams also ship without coordinating releases.

Cost: every internal call becomes a network call. You inherit partial failure, distributed tracing, versioned contracts, and the availability multiplication from Lesson 2 — four synchronous services at 99.9% give you 99.6%.

Best practices

Beyond the big architectural moves:

Practice	What it means	Typical win
Mitigate bottlenecks	Find and fix inefficient queries and algorithms	Often the largest single gain — a missing index beats a new server
Efficient resource use	Queues for requests, workers for background jobs, caching for repeats	Keeps the interactive path fast under load
Minimize network latency	Fewer hops, caching, optimized transfer protocols	Directly cuts p99 (Chapter 1's fallacy 2)
Optimize data storage	Scalable storage patterns, efficient indexing, replication, partitioning	Keeps reads fast as data grows
Select appropriate technology	Efficient algorithms, tuned queries, hardware fit for the workload (e.g. SSDs)	Avoids solving with hardware what a data structure would fix

Cloud computing makes this easier by allowing resources to be added or removed on demand, so traffic spikes and growth don't require physical hardware upgrades. DevOps practices streamline scaling further by automating infrastructure management and enabling faster deployments.

Challenges and trade-offs

Scaling introduces costs that grow with the system:

Challenge	Why it grows	Mitigation
Cost	More resources, more redundancy, more egress	Right-size, autoscale, tier storage by access frequency
Consistency	Data spread across nodes diverges	Choose models per data type (Chapter 1); session guarantees where they suffice
Security	More services means a larger attack surface and more policy to enforce	Zero trust, authn/authz per hop, centralized policy
Complexity	Distributed systems are harder to manage, debug, and troubleshoot	Observability, tracing, and refusing complexity you don't yet need

Real-world examples

Google Search — a massively scalable architecture processing billions of daily queries.
Netflix — cloud infrastructure (AWS) handling millions of concurrent streams.
Facebook — requests and data from billions of global users.
Uber — millions of ride requests globally, in real time.

The common pattern across all four: caching and CDNs at the edge, load balancing across stateless tiers, sharded and replicated storage, and asynchronous processing for anything a user isn't waiting on.