Workflow, Concurrency, and Custom Aliases

In one line: the deduplication step here quietly reintroduces the database read that Lesson 6 said the sequencer design had eliminated — and the used/unused ID tracking is the price of custom aliases.

The four operations

Deduplication

Computing a short URL for an already existing long URL is redundant. The system sends the long URL to the database server to check its existence. It checks the cache first, then queries the database.

If a short URL for that long URL already exists, the database returns it. If not, the application server asks the generator to compute one.

This puts a read back on the write path — the thing the sequencer design avoided

Lesson 6 made a point of it: a sequencer produces unique IDs without checking, unlike a hash-based scheme that needs a collision lookup on every write.

Deduplication reintroduces exactly that read — for a different reason, but with the same cost:

Hash-based:  write -> READ to check for COLLISION -> write
Sequencer:   write -> READ to check for DUPLICATE LONG URL -> generate -> write

At 76 writes per second this is completely affordable. But it is worth being honest that the design's cleanest property is preserved only for the generation step, not for the operation.

And the read is not free architecturally. It requires a secondary index on the long URL — a second index on a 12-billion-row table, used at 76 QPS while the primary index serves 7,600. A hundredfold difference in access frequency between two indexes on the same table, which Lesson 5 noted the schema discussion never mentions.

There is also a question the design never asks: is deduplication even desirable?

Since the resources it saves are both abundant, the argument for deduplication is weak — and the argument against is real: two marketing campaigns shortening the same landing page probably want separate links so they can measure them separately.

A deduplication step that saves an abundant resource and destroys a useful distinction is worth questioning. Most commercial shorteners do not deduplicate.

Canonicalizing before storing is what makes deduplication meaningful: example.com/x, example.com/x/ and EXAMPLE.com/x are one destination, and treating them as three entries wastes ID space and defeats the point.

Concurrency

1. MongoDB ensures consistency by locking and concurrency control protocols.
2. All write requests go through the single leader, excluding race conditions due to the serialization of requests.

Serialization through one leader is the whole concurrency answer, and it is sufficient here

Lesson 5 established why this works: at 76 writes per second, a single serializing writer is not a bottleneck, and serialization gives correctness for free.

Two races it closes:

Two users claiming the same custom alias. Serialized, so the second sees the first's write and gets a duplicate-key error.

Two requests shortening the same long URL simultaneously. Both check for an existing mapping, both find none, both generate. Serialization plus a unique index on the long URL means the second insert fails and can return the first's result.

That second case is the interesting one, because it shows deduplication needs more than a lookup — a check-then-write has a window between the two steps. The unique index is what actually closes it; the lookup is just an optimization to avoid the common case reaching the error path.

A read-then-write check is never sufficient on its own; the constraint that rejects the duplicate is what provides the guarantee.

Custom aliases and the used/unused lists

The server decodes the custom short URL back to its base-10 equivalent and marks that ID as "used" in the database. This ensures no two long URLs can map to the same short URL.

Newly generated IDs are added to an "unused" list. Once an ID is assigned, it is moved to the "used" list. Because base-58 encoding provides a one-to-one mapping, it prevents collisions.

An 'unused list' over a 64-bit space cannot be materialized — and it does not need to be

Read the description literally and it is impossible. There are 1.8 × 10¹⁹ IDs; you cannot store a list of them.

The workable reading is that "unused" means "generated but not yet assigned" — a small buffer of pre-minted IDs, not an enumeration of the space. The chapter's phrasing, "once we generate IDs, we put them in the unused list," supports this.

So the real state is:

UNUSED:  a small pool of pre-generated IDs, ready to hand out
USED:    IDs that have been assigned  ->  in practice, this IS the URL table

And the "used list" is redundant with the mapping table itself: if an ID is in the URL table, it is used. Maintaining a separate list means keeping two things consistent for no benefit.

The only case that genuinely needs extra bookkeeping is a custom alias whose decoded ID the sequencer has not yet reached — you must record it so the sequencer skips it later. And even that is unnecessary if the sequencer checks the URL table before assigning, which it must do anyway to be safe.

A separate used/unused structure is redundant with the table that already records assignments. The chapter's stated benefit — "the node generating short URLs will no longer need to maintain a list in memory" — is real and achieved simply by consulting the database, without a second data structure.

Custom aliases and generated IDs are competing for one namespace

This is the deeper structural problem, and the design's mechanism is a symptom of it.

Both sources write into the same space of short strings:

SEQUENCER  -> integer -> encode -> string  \
                                            > SAME NAMESPACE
USER       -> string  -> decode -> integer /

Which is why the decode step exists at all (Lesson 8) and why used/unused tracking is needed.

An alternative the design does not consider: separate the namespaces.

Generated:  6+ characters, drawn from the sequencer range
Custom:     a RESERVED prefix or a distinct length class

If custom aliases lived in a space the sequencer never touches — say, all aliases containing a specific marker, or a length the generator never produces — then:

• No decoding needed. The bijection is only required in one direction again.
• No used/unused tracking. The sequencer cannot collide with something it cannot generate.
• Custom-alias collisions reduce to a plain unique-index check on a string.

The cost is that custom aliases become slightly less free-form. Given Lesson 10's finding that user-chosen aliases are predictable by construction anyway, that seems a small price.

When two allocators write into one namespace, either they must coordinate or the namespace must be partitioned — and partitioning is usually simpler.

Next: running this across multiple data centres.