APIs and Storage Schema

In one line: the schema is small and one column in it — heartbeat — is doing all the fault-tolerance work. It is also where the design's most subtle bug lives.

Build APIs

submitCode(code, repository, branch, message)      -> returns a commit hash
triggerBuild(project_id, branch, commit_hash, configuration)
                                                   -> returns {queue_url, build_id}, ASYNCHRONOUSLY
accessBuildArtifacts(build_id)
debugBuildFailures(build_id)                       -> logs, stack traces, error messages, context

triggerBuild returning immediately is the right call, and the return value tells you why

"triggerBuild() initiates the build process asynchronously and returns immediately. It provides a JSON object containing the queue URL and the assigned build_id."

A build takes twenty minutes. A synchronous API would mean a connection held open for twenty minutes, with a client that must handle its own timeout, retry semantics that are ambiguous (did the retry start a second build?), and no way to check progress.

Returning a build_id immediately converts a long-running operation into a resource the client can poll or subscribe to — the standard pattern for anything slower than a few seconds. The queue URL is the nice touch: it tells the caller where their job is, so progress is observable without a separate status endpoint.

Any operation whose duration exceeds a request timeout should return a handle, not a result. The same shape appears in that building block's own deployment APIs, and it is what makes monitorDeploymentProgress() possible at all.

debugBuildFailures is an unusual API to see specified, and its presence is a signal

Most designs specify the happy path. This one gives failure diagnosis a first-class endpoint returning "stack traces, error messages, code snippets, and related context."

That belongs here because of who the users are. A deployment system's users are engineers, and the single most common thing an engineer does with a build system is find out why the build broke. Optimizing for that is optimizing for the dominant use case, not for an edge case.

It also connects to the fault-tolerance requirement from Lesson 3: "every build must definitively report success or failure." A definite failure is only useful if you can then find out what failed — the two requirements are one requirement.

Design the failure path with the same care as the success path when your users are the people who debug it.

Deployment APIs

selectTargetEnvironment(environment)               -> production | staging | development | custom
initiateDeployment(build_id, build_version, strategy)
monitorDeploymentProgress(deployment_id)
viewDeploymentLogs(deployment_id)
manageRollback(deployment_id)
validateAndTest(deployment_id, environment)

validateAndTest exists in the API list and nothing in the design ever calls it

This is worth flagging now because it recurs.

validateAndTest(deployment_id, environment) is specified as "triggers validation and testing on the deployed application." It corresponds to stage 4 of the seven-stage pipeline from Lesson 1.

Then trace the detailed design's workflow:

build -> primary blob -> replication -> regional blob -> config update
      -> servers poll -> download -> install -> status "up"

There is no validation step anywhere in that sequence. No test gate between build and deploy, no health check gating promotion, no staging environment. The API exists; the workflow never invokes it.

The same is true of selectTargetEnvironment(environment) — it accepts staging, and no staging environment appears anywhere in the architecture.

An API that nothing calls is a requirement that was listed and not designed. Lesson 8 covers what filling that gap requires; for now, note the pattern, because "the API list is more complete than the workflow" is a common shape in system design write-ups and a good thing to catch.

manageRollback(deployment_id) is missing the parameter that matters

Look at what the signature does and does not say:

manageRollback(deployment_id)     <- roll back THIS deployment

It identifies what to undo. It does not say what to roll back to. And "the previous version" is ambiguous in exactly the situation where you need rollback most:

v1.0 deployed Monday       -> known good
v1.1 deployed Tuesday      -> subtly broken, nobody noticed
v1.2 deployed Wednesday    -> obviously broken
  -> rolling back v1.2 gives you v1.1, which is ALSO broken

The compliance table describes rollback as "the configuration service can update the deployment version (e.g. build:1.0) to a previous one" — which is a target version, not a deployment ID. So the mechanism supports what the signature doesn't express.

The API should be manageRollback(deployment_id, target_version), or better, deployVersion(target_version) — because a rollback is just a deployment of an older artifact, and treating it as its own operation with its own code path means the rarely-used path is the one you need under pressure.

Make rollback the same code path as deployment, with a different argument. A separate mechanism is a mechanism that is only exercised during incidents.

Storage schema

Persisting the queue in a database is a deliberate choice, and the reason is the fault-tolerance requirement

The design is explicit: "The database schema must support the API attributes and persist jobs in the pub-sub queue to ensure a reliable build process."

A queue already holds jobs. Why write them to a table as well?

Because a message queue tells you what is waiting; it does not tell you what is running. Once a worker takes a message, the queue's view of that job is gone — and the requirement is that every build reports success or failure, including builds whose worker died mid-run.

The QueueTable gives you a durable, queryable record of every job's lifecycle:

Queued    -> placed in the queue
Running   -> assigned to a build worker
Failed    -> technical error
Completed -> artifacts in the blob store
Cancelled -> abandoned

So the table is not a duplicate of the queue; it is the job's state machine, which the queue alone cannot represent. And it is what makes the heartbeat mechanism possible — you cannot detect a stalled job in a queue that has already handed it out.

When "no job may be lost or left in an unknown state" is a requirement, the queue is not enough — you need durable per-job state alongside it.

The heartbeat

The worker updates this attribute regularly to indicate that it is active. If the heartbeat is not updated within the expected interval, the system assumes the worker has failed. The job status is reset to Queued, allowing it to be reassigned to another worker.

This is a lease, and leases are how distributed systems detect death without asking

The pattern generalizes far beyond builds, and naming it is worth doing.

You cannot ask a dead process whether it is dead. So instead of detecting failure, you require continuous proof of life — a lease the worker must keep renewing. Stop renewing, for any reason, and the lease expires and the work is reclaimed.

The elegance is that it handles every failure mode identically: process crash, machine loss, network partition, kernel panic, someone tripping over a cable. None of them can renew a heartbeat, so all of them look the same to the system and all are recovered the same way.

The module has met this before — ZooKeeper's ephemeral nodes, the monitoring chapter's health checks, primary-election leases. Every distributed system that assigns exclusive work to a worker needs a lease, because there is no other way to distinguish "slow" from "dead."

Except that last part is exactly the problem.

The lease can reassign a job that is still running — and nothing here prevents the duplicate

The mechanism as described has a gap that every lease-based system has to close, and this one does not.

A slow worker and a dead worker are indistinguishable. A long garbage-collection pause, a saturated network link, a stalled disk — any of these can delay a heartbeat past its interval on a worker that is very much alive and mid-build.

What happens next:

Worker A: building commit abc123, heartbeat delayed 40s by GC pause
System:   heartbeat stale -> status = Queued
Worker B: claims job, starts building commit abc123
Worker A: GC finishes, resumes, finishes the build
  -> TWO workers building the same commit
  -> BOTH write artifacts to the same blob path

For a build this is less catastrophic than for a payment — builds are close to idempotent, so the two artifacts are probably equivalent. But "probably" is doing real work in that sentence: builds that embed timestamps, build numbers, or resolve dependency ranges are not reproducible, so the two artifacts can genuinely differ. And whichever finishes last wins, silently.

The standard fix is a fencing token — a monotonically increasing number issued with each lease:

Worker A holds lease 17.  Worker B is granted lease 18.
Every write to blob storage includes the token.
The store rejects any write with a token lower than the highest it has seen.
  -> Worker A's late write is REJECTED. Exactly one artifact survives.

Note what fencing does and does not do: it does not prevent duplicate work — both workers still burn CPU — it prevents duplicate effects. That is the achievable goal. You cannot prevent duplicate execution in a distributed system; you can only make the duplicate's side effects harmless.

This is the same conclusion that building block reached about idempotency, and it will return in that building block, where the side effect is charging someone's card and "probably equivalent" is not acceptable.

A lease without a fencing token is a lease that guarantees at-least-once execution and nothing about the writes.

Two schema details worth noticing

sha appears in all three tables. It is the commit hash, and it is the join key that ties a queue job to a build to a deployed binary. That is the right choice — the commit is the stable identity of a change, and everything else (build IDs, deployment IDs, binary IDs) is generated. When an incident starts with "what shipped?", the answer is a SHA.

heartbeat: VARCHAR is the wrong type. A heartbeat is a point in time and belongs in a TIMESTAMP, exactly like created_at in the same table. As a string, "is this stale?" becomes a parse-then-compare instead of an indexed range query — and the query that finds stale jobs runs constantly, across every running job. Minor, but it is the kind of thing that turns into a table scan at scale.

Next: the build phase.