Secondary Indexes and Request Routing

Why this matters: partitioning works beautifully until someone asks a question that doesn't mention the partition key. Then you choose between slow reads and slow writes — there is no third option.

Key takeaway

Partitioning by primary key handles lookups by that key. Secondary indexes let you query by other attributes, and partitioning them forces a choice: pay on reads (local index) or pay on writes (global index).

The problem

So far, data is partitioned by primary key. But applications routinely need to query by other attributes — searching for customers by creation year, or by name. Those attributes say nothing about which shard holds the row.

There are two ways to partition secondary indexes.

Partition by document (local index)

Each shard maintains its own secondary index, covering only the data in that shard. To write, you only need to touch the partition containing the document being written.

Trade-off: writes are fast because they only update the local shard. Reads are expensive because the system must query every partition — a scatter-gather — and merge the results. If one partition is slow, the entire query is delayed.

Partition by term (global index)

A single global index covers data from all partitions, and the index itself is partitioned by the indexed term.

Index 0 might store names starting with A-M and Index 1 names N-Z. To find a customer named "John", the client queries only Index 0.

Trade-off: reads are efficient because they target specific nodes. Writes are complex and slow because a single record update might require updating indexes on multiple different nodes — a distributed transaction.

The comparison

	By document (local)	By term (global)
Index scope	One index per shard, covering that shard only	One global index, partitioned by term
Write cost	Cheap — one shard	Expensive — possibly several index nodes
Read cost	Expensive — scatter-gather across all shards	Cheap — targets specific nodes
Consistency	Index always matches its shard	Index may lag; often updated asynchronously
Reach for it when	Write-heavy, few shards, or queries already filtered by partition key	Read-heavy, many shards, latency-sensitive lookups

Request routing

How does a client know which node to connect to? This is the service discovery problem, and there are three standard answers.

Strategy	How it works	Trade-off
Any node	Clients connect to any node; if it doesn't hold the data it forwards to the correct one	Simple clients; costs an extra hop
Routing tier	A dedicated routing layer forwards requests to the appropriate node	Centralized logic; another component to run and scale
Client-aware	The client maintains the partitioning map and connects directly	Fastest — no extra hop; but routing logic ships in every client

The challenge in all three approaches is the same: keeping routing information up to date as the topology changes. Nodes are added, removed, and rebalanced, and a stale map sends requests to the wrong place.

ZooKeeper

Distributed systems often use a coordination service to track cluster state. ZooKeeper maintains the mapping of partitions to nodes. When a node is added or removed, ZooKeeper updates the mapping and notifies the routing tier or clients. Systems including HBase, Kafka, and SolrCloud rely on it.