Execution Caps, Delay Tolerance, and Capacity

In one line: these three parameters are how the scheduler is actually tuned. The execution cap has a genuinely hard problem inside it — you cannot distinguish a bug from slow legitimate work by looking at elapsed time.

Execution cap

Clients can specify a time limit for their tasks. If a task exceeds this cap, the scheduler terminates it, releases the resource, and notifies the client. If no cap is set, the system applies a default upper bound.

The hard part: a timeout cannot tell a bug from slow work

This is the honest difficulty, and the design names it: "distinguishing between a faulty task and a legitimately long-running workload is non-trivial."

From outside the process, an infinite loop and a large ML training run look identical — both consume CPU and do not finish. There is no signal in elapsed time that separates them, so any timeout is a guess, and both errors are costly:

• Cap too low → you kill legitimate work, wasting everything it computed and frustrating users.
• Cap too high → runaway tasks hold resources for hours, which is exactly the monopolization you were preventing.

The design's answer is to push the judgment to the client: they specify the cap because only they know what their task should take, with a default upper bound as a backstop for those who do not.

That is the right call — the scheduler genuinely lacks the information — and it is worth naming as such rather than presenting the cap as a solution. It relocates the judgment to where the knowledge is.

The better long-term signal is progress, not time: a task reporting checkpoints is demonstrably working, while one that has reported nothing for an hour is suspect regardless of its cap. That requires cooperation from the task, which brings us to checkpointing.

Checkpointing and the cost of termination

For legitimate long-running tasks, such as training machine learning models, the scheduler may pause and resume execution to accommodate urgent work. Clients should implement checkpointing to periodically save state, allowing the task to resume its progress after an interruption.

"What if a long task is 90% executed, but the machine executing it fails?"

The task scheduler will re-execute the task on some other machine. Tasks need to be either idempotent or able to restore their state from a previous checkpoint. Once the state is saved, we can resume execution of that task on any other machine. This makes our system fault-tolerant and saves resources.

Notice where checkpointing lives — the application, not the scheduler

The design says clients should implement checkpointing. That is a deliberate boundary, and it is worth being explicit about the contract it creates:

The scheduler promises: your task will eventually run to completion, retried on a healthy machine if necessary.

The scheduler does not promise: that any partial work survives. It will happily discard 90% progress and start over.

Preserving progress is the application's job, because only the application knows what its state is and how to serialize it. A scheduler cannot generically checkpoint arbitrary code — that would require process migration, which is far harder and is why Lesson 3 assumed short-lived tasks in the first place.

So the design's position is coherent: short tasks make rerunning cheap, and anything long enough for rerunning to hurt must checkpoint itself. State the boundary clearly in an interview — candidates often assume the scheduler magically preserves progress.

Checkpointing enables preemption, which is what makes priority useful mid-flight

The pause-and-resume clause matters more than it appears. Without checkpointing, an urgent task arriving while all resources are busy has two options: wait, or kill a running task and lose its work.

With checkpointing, there is a third: preempt — save the running task's state, free the resource for the urgent task, and resume the preempted one later.

That is what makes Lesson 6's priority tiers meaningful during execution rather than only at dispatch. Without preemption, a task that has already started is effectively immune to priority until it finishes, so an urgent arrival waits for whatever is running regardless of its tier.

Lesson 1 noted that preemption in a data center costs everything computed so far, unlike an OS context switch. Checkpointing is what brings that cost back down to something a scheduler can afford to do.

The execution cap and checkpointing are the same decision seen twice: the cap bounds how long a task may hold a slot, and checkpointing bounds how much is lost when the cap fires. Without checkpoints the cap is a guillotine; with them it is a pause.

Prioritization by delay tolerance

The scheduler assigns a delay tolerance to each task, defining the maximum acceptable wait time. It prioritizes tasks with the shortest delay tolerance, postponing flexible tasks to ensure urgent ones meet their deadlines.

"How do we determine the value of delay tolerance?"

Application owners or clients can set or automate values themselves, depending on the task category. In a social media application: generate a newsfeed, suggest friends, allow users to mark themselves safe after a disaster, send notifications about a live stream event. Among these, the top priority is marking a person safe during an earthquake and sending live stream notifications. Clients can tighten delay tolerance down to milliseconds or a few seconds, while tasks like suggesting friends can be delayed for days.

There are different costs for different priorities — higher costs for high-priority tasks — so customers can carefully categorize their tasks.

Delay tolerance is derived from the business requirement, not chosen from a menu

Compare the two framings:

"What priority is this task?" — everyone answers high. Priority is free to claim, so the field carries no information.

"How long can this wait before it stops being useful?" — has a real answer grounded in what the task does. Marking yourself safe during an earthquake: seconds. Friend suggestions: days.

The second question is answerable and hard to inflate, which is why it is the better field. The schema in Lesson 5 has no priority column — priority is computed from delay tolerance, which is exactly right.

And note the enforcement mechanism: different costs for different priorities. That is the only thing that makes self-declared urgency honest. Without price attached, every task is urgent; with it, customers do the categorization themselves because inflating urgency costs them money.

Price is what makes a self-reported field truthful. That is a genuinely transferable insight — the same reason cloud providers charge more for guaranteed capacity than for spot instances.

Resource capacity optimization

Resources often experience peak time loads (e.g. >80% utilization) while remaining idle during off-peak hours. Non-urgent tasks, such as friend suggestion generation, can be deferred and scheduled during periods of lower system load.

Cloud platforms continuously monitor capacity utilization relative to workload demand. When demand rises, the system scales out by provisioning additional instances. When demand falls, it scales in to reduce infrastructure costs.

Utilization over a day

    peak                          peak
     ###                           ###
    #####                         #####
   #######      trough           #######
  #########   ~~~~~~~~~~~~~~~  #########
 -----------------------------------------
             ^ deferrable work fills this

Deferrable work is free capacity — this is the strongest economic argument in the chapter

The insight is that a fleet sized for peak is idle the rest of the time, and that idle capacity is already paid for.

Delay tolerance gives the scheduler exactly what it needs to exploit this: tasks that can wait days — friend suggestions, batch analytics, index rebuilds — should not run at peak at all. Push them into the trough and they cost nothing extra, because those machines exist and are running regardless.

So the scheduler is doing something better than ordering work. It is shaping demand to fit supply, which is only possible because tasks carry an explicit statement of how long they can wait.

Two consequences worth stating:

It reduces peak provisioning. If deferrable work is pushed out of peak, the fleet can be sized for genuinely urgent load, which is smaller.

It is why the cost tiers make sense. Low-priority tasks are cheaper because they genuinely cost less to serve — they run on capacity you already have. That is not a discount; it is accurate pricing.

This is the same idea as object storage's access tiers: match the workload's flexibility to the resource's cost profile.

Scaling in and out interacts awkwardly with long tasks

"When demand falls, it scales in" is straightforward for stateless web servers and genuinely awkward here: which machine do you remove when tasks are still running on it?

Three options, none free. Wait for the task to finish — correct, but drain time is bounded by your execution cap, so scale-in is slow. Kill and reschedule — fast, but wastes work unless the task checkpointed. Preempt with checkpointing — best, and requires application cooperation.

So the same checkpointing that enables priority preemption also enables efficient scale-in. It is worth noticing how much of this design's flexibility traces back to that one application-level capability — and that the scheduler cannot provide it for you.

Interview signal by level

Next: why retries make idempotency mandatory.