Exactly-Once, Dual Writes, and the Outbox

In one line: the design's transaction-completion mechanism is correct about durability and wrong about duplication — and it writes to two databases with nothing making them atomic.

What the design specifies

To ensure transaction completion, we can use a pub-sub system such as Apache Kafka. When a payment is initiated, we publish an event to a Kafka topic. Consumer services process these events, and a message is only marked as consumed after the transaction is successfully processed and recorded in the wallet and ledger databases. This ensures that no payment events are lost.

Ack-after-processing is the right pattern, and it delivers exactly the guarantee it names

The mechanism is correct and worth stating precisely: acknowledge the message only after the work is durably done.

If the consumer crashes before acknowledging, the broker has no record of completion and redelivers. So a payment event cannot be silently dropped by a consumer failure — which is exactly the claim the design makes: "no payment events are lost."

Contrast with the alternative:

ACK FIRST, then process:   crash mid-processing -> the event is GONE
                           -> at-MOST-once. Payments vanish.
ACK AFTER processing:      crash mid-processing -> the event is REDELIVERED
                           -> at-LEAST-once. Nothing is lost.

For a payment system, losing an event is unacceptable, so at-least-once is the right choice.

The claim is accurate. The problem is what it does not claim, and what the design does with the consequence.

At-least-once is not exactly-once

Redelivery plus a mutable balance is a double credit

Ack-after-processing guarantees at least once. It explicitly permits more than once, and the window is unavoidable:

There is no way to close that window by ordering the operations differently. Committing the write and acknowledging the message are two separate actions against two separate systems, and a crash can land between them in either order:

Write then ack:  crash between -> the write happened, redelivery repeats it
Ack then write:  crash between -> the write never happened, and it's gone

Pick either and one failure mode remains. This is the same impossibility that building block reached about exactly-once message delivery, and that building block reached about duplicate build execution.

The difference is what a duplicate costs. that building block's duplicate produced "probably equivalent artifacts." Here it produces a merchant credited twice for one payment, and nothing in the system detects it — because, per Lesson 3, the balance is a mutable column with nothing to check it against.

The design applies idempotency to the client retry path and not to the consumer, which is where duplicates are far more frequent: consumer redelivery happens on every deploy, every rebalance, every broker failover, every crash.

At-least-once delivery plus a non-idempotent handler equals corruption. The delivery guarantee is only half the design; the handler must supply the other half.

The fix is the same idempotency key, moved down a layer

Exactly-once delivery is not achievable. Exactly-once effect is, and it needs no new machinery — just the mechanism from Lesson 6 applied at the consumer.

Carry the idempotency key on the event, and let the database reject the duplicate:

Consumer receives event with key "pay_7f3a"
  BEGIN
    INSERT INTO ledger_entry (idempotency_key, account, amount_minor, ...)
      VALUES ('pay_7f3a', ...)         -- UNIQUE constraint
    -- if this violates uniqueness: already applied. Roll back, ack, done.
    UPDATE account_balance SET balance_minor = balance_minor + 10000 ...
  COMMIT
  ack

Redelivery now hits the unique constraint, the transaction rolls back having changed nothing, and the consumer acknowledges. The duplicate is absorbed by the database rather than by a check in application code, which is what removes the race.

Notice this is only safe because the ledger design from Lesson 5 exists. Against the design's schema — a Ledger with no amount and a Wallet with a mutable balance — there is no natural place to put a unique constraint and nothing to make the two writes atomic.

Idempotent consumers are what turn at-least-once delivery into exactly-once effect, and the constraint has to live in the same transaction as the state change.

The dual-write problem

Wallet and ledger are two databases, and a crash between them leaves them disagreeing

Read the design's sentence again: "recorded in the wallet and ledger databases." Two stores. The consumer writes to both.

This is the dual-write problem, and it is one of the most common sources of data corruption in service architectures. Two writes to two systems cannot be made atomic by ordering them, because there is no transaction spanning both:

Wallet then ledger:  crash between -> money credited, no audit record
Ledger then wallet:  crash between -> audit record, money not credited

And here the consequence is specific: the reconciliation system will eventually flag a mismatch it cannot explain. The chapter describes reconciliation as catching "missing transactions, duplicate entries, timing delays, or data corruption" — this is the design generating exactly that corruption itself, and reconciliation has no way to attribute it.

Two fixes, in order of preference:

Make it one write. Put the ledger entries and the balance in the same database, updated in a single transaction. The balance is a materialized view of the ledger (Lesson 5), so keeping them together is natural rather than a compromise. This eliminates the problem instead of managing it, and at 1,400 TPS there is no scale argument for splitting them.

Or use a transactional outbox. Where the second system genuinely must be separate, write the state change and an outbox row in one local transaction, then have a relay publish from the outbox:

BEGIN
  INSERT INTO ledger_entry ...
  UPDATE account_balance ...
  INSERT INTO outbox (event) VALUES (...)     -- same transaction
COMMIT
-- a separate relay reads outbox -> publishes -> marks sent

Now there is exactly one atomic write, and publication is derived from committed state rather than racing it. The relay is at-least-once, which is fine because consumers are idempotent.

Never write to two systems and hope both succeed — make one write atomic and derive the other from it.

Ordering

Payments for one account must be ordered, and the design doesn't say so

Kafka guarantees ordering within a partition, and nothing in the design specifies a partition key. Without one, events distribute round-robin and a merchant's payments can be processed out of order across consumers.

For a pure ledger of appends, order barely matters — addition commutes, so entries applied in any order sum to the same balance. But it matters for anything conditional:

Event 1: charge  $100 to merchant M
Event 2: refund  $100 from merchant M

Applied in order:            balance +100 then -100 -> 0, never negative
Applied out of order:        balance -100 first -> a negative balance,
                             which may trip a solvency check or a payout block

And it matters absolutely for state transitions — an Authorized → Captured event applied before the Authorized event that created the transaction has nothing to act on.

The fix is the standard one and it is the same insight as that building block's per-document queues: partition by the unit within which order must hold.

Partition key = merchant_id (or account_id)
  -> all events for one account land in one partition, strictly ordered
  -> different accounts process fully in parallel

Order only needs to be global within the entity that shares state, and for a payment system that entity is the account, not the system.

The three guarantees people conflate

Worth having straight, because interviewers ask and the terms are used loosely:

At-most-once   ack before processing.   Messages can be LOST. Never for payments.
At-least-once  ack after processing.    Messages can be DUPLICATED. The right choice.
Exactly-once   NOT achievable end-to-end across independent systems.

What is achievable is exactly-once effect: at-least-once delivery plus idempotent handling. The duplicate still arrives; it just changes nothing.

That is the same conclusion every chapter in this module has reached from a different direction — that building block through commutativity and idempotency of edit operations, that building block through fencing tokens on a lease, and here through unique constraints on ledger entries.

Nobody achieves exactly-once delivery. Everybody achieves exactly-once effect, by making the second attempt do nothing.

Change data capture: the other way out of the dual write

The transactional outbox solves the dual-write problem by making the message part of the same transaction as the data. There is a second approach worth knowing, because an interviewer may name it.

CDC reads the database's own write-ahead log — the record the database already keeps to make commits durable — and turns each committed change into a stream event. Because it derives from the log rather than from application code, an event exists if and only if the transaction committed, which is exactly the guarantee the dual write breaks.

The trade against the outbox:

Prefer CDC when you want the audit trail to be complete by construction, which is the payment case: an event stream derived from the log cannot omit a committed transaction, no matter what the application forgot to publish. Prefer the outbox when you want to publish a curated domain event rather than a row diff.

Both deliver at least once, so the consumer still needs to be idempotent — neither removes that obligation, they only remove the possibility of a missing event.

Next: retries, timeouts, and a fallback that fails open.