The Architecture

In one line: this is the diagram to be able to draw. The two details worth arguing for are the dual database — relational plus graph — and the task submitter as a cluster with its own failure handling.

What a task submission contains

The design

The dual database is the design's sharpest choice — relational for metadata, graph for the DAG

Two stores for two genuinely different shapes of data:

Relational holds task metadata — IDs, resource requirements, execution caps, retry counts. This is structured, uniform, and queried by attribute: "give me the top K tasks by delay tolerance." That is a SELECT ... ORDER BY ... LIMIT, exactly the relational sweet spot the databases material described.

Graph holds the directed acyclic graph of dependencies, so the system can do a topological sort to determine a valid execution order.

Why not put dependencies in the relational store? You could — a prerequisites table — but the queries you need are traversals: "what is now unblocked given that task 47 finished?", "what is the full transitive closure of what this task blocks?" Those are recursive joins in SQL, and they get expensive fast. Graph databases exist because traversal is their primitive operation.

The general principle, and it recurs: choose the store by the shape of the query, not by the shape of the data. The Blob Store chapter split metadata across relational and key-value stores for exactly the same reason.

One thing to note: acyclic is load-bearing. A cycle in the dependency graph means tasks waiting on each other forever — a deadlock — so the system must reject cyclic submissions at admission time, which is a cheap check the graph database makes easy.

'Top K, where K is determined by available resources' is backpressure done properly

The batching layer does not push everything it has into the queue. It pushes K tasks, and K is set by available resources and subscription levels.

That is a real design decision. Pushing everything would make the queue an unbounded buffer, and Lesson 3 noted the queue then just relocates the problem — tasks would sit in the queue rather than the database, with no benefit and worse visibility.

Keeping most tasks in the database and promoting only a working set gives three things:

• Reprioritization stays possible. A task still in the database can have its priority recomputed as conditions change. Once in the queue, its position is largely fixed.
• Cancellation is easy — Lesson 2's "remove tasks" requirement is a database delete, not a queue surgery operation.
• The queue stays short, so queue operations stay fast and its state stays comprehensible.

This is the same insight as distributed caching's working set: hold everything in cheap storage, promote only what you are about to need.

The queue manager and resource manager split the work by lifecycle stage

Two managers, and the boundary is clean:

Queue manager owns the task before and around execution — visibility, deletion on success, retry limits on failure, and which queue to use given load. It is message-queue design's visibility timeout mechanism, reused wholesale: receive makes the task invisible, success deletes it, failure makes it visible again.

Resource manager owns the task during execution — assigning resources, watching status, and terminating anything exceeding its limits.

That split matters because they fail differently. A queue manager failure means tasks are not dequeued — bad, but nothing is lost, since the queue persists them. A resource manager failure means running tasks are unmonitored — a task in an infinite loop holds its machine indefinitely, which is why Lesson 7's execution cap exists and why monitoring watches the resource manager itself, not just the resources.

Who decides a task is due?

The scheduler runs as a cluster for availability, which creates a question the single-node picture hides: if five scheduler nodes all notice the same task is due, does it run five times?

Two workable answers, and the choice is about scale rather than correctness.

Leader election. One node holds a lease from a coordination service and is the only one permitted to dispatch. Simple and obviously correct, and the leader becomes both a throughput ceiling and a small availability gap during failover — the lease must expire before another node can take over, so there is a window where nothing dispatches.

Partitioned ownership. Each node owns a slice of the task space — by task id hash, or by tenant — so two nodes never consider the same task. That scales linearly and moves the difficulty into the partition map and into handoff when a node dies.

Whichever you choose, the claim itself must be atomic: a node marks the task as claimed with a conditional write, and only the winner executes. Belt and braces, because leases expire and partition maps go briefly stale, and the conditional claim is what makes those windows harmless.

Two clocks that disagree

Time-triggered scheduling has a dependency most designs never mention: the correctness of the clock.

Synchronized clocks reduce the window; they do not close it. The scheduler should treat time as a trigger to attempt a claim, not as authorization to execute — which is the same reasoning as the atomic claim above, and why "we run NTP" is not on its own an answer.

The task submitter cluster

To ensure high availability, the task submitter operates as a cluster of nodes. Each node admits tasks, requests IDs, and writes to the database.

A cluster manager monitors these nodes via heartbeats. It maintains a mapping of tasks to the admitting node. If a node fails, the cluster manager reassigns its tasks to a healthy node. The cluster manager itself is replicated to prevent a single point of failure.

The task-to-node mapping exists because admission is not atomic

Why track which node admitted which task? Because admission is several steps — get an ID, write metadata to the relational store, write dependencies to the graph store — and a node can die partway through.

Without the mapping, a node dying mid-admission leaves a task in limbo: the client was told it was accepted, but the record may be incomplete and nothing knows to finish it. That is Lesson 1's silent failure exactly — the user believes work was accepted that will never run.

The mapping lets the cluster manager identify the orphans and reassign them to a healthy node, which completes or restarts admission.

Note the acknowledgment that the cluster manager is itself critical: "the cluster manager itself is replicated to prevent a single point of failure." Same conclusion as distributed caching's configuration service and object storage's manager node — the component that tracks cluster state needs its own replication, or you have relocated the single point of failure rather than removed it.

Geo-replication lets multiple schedulers run independently

Note: using geo-replicated data stores allows multiple instances of the scheduler to run in different data centers, improving scale and utilization.

This answers Lesson 1's second challenge — resources dispersed across data centers. Rather than one global scheduler making every placement decision (a bottleneck, a single point of failure, and a cross-region round trip per decision), each data center runs its own scheduler instance against geo-replicated shared state.

The trade is the familiar one from the Rate Limiter and Distributed Cache chapters: replicated state across regions is asynchronous, so two schedulers can briefly hold different views of what is free and both assign the same resource.

That is survivable here in a way it would not be elsewhere — the resource manager detects the conflict at assignment time, and one task simply goes back to the queue. Cheap conflicts plus retry beats expensive global coordination, which is precisely what Lesson 3's short-lived-task assumption buys.

Interview signal by level

Next: what the database actually stores.