Free preview

Why this matters: the pool you built is one point in a large design space, and interviewers move you around that space to see whether you understood your own choices. Every variation below lands against a seam from lesson 02 — and knowing which seam absorbs which change is the difference between evolving a design and re-deriving one under pressure.

Sizing: the question every caller asks

Fixed-N was a requirement; choosing N is reasoning you should be able to do out loud.

CPU-bound tasks — image transforms, compression, scoring — want roughly one worker per core. More workers than cores can't add throughput; they just make the OS referee context switches between threads that all want the same silicon.

IO-bound tasks — network calls, disk reads — spend most of their life waiting. A worker blocked on IO occupies a pool slot while using no CPU, so the pool wants more workers than cores, scaled by the waiting fraction: if a task waits 90% of the time, ten workers keep roughly one core busy. The cleaner architectural answer, worth naming: don't share one pool between the two workloads at all. A shared pool's IO tasks can occupy every worker while CPU work starves behind them — separate pools with separate sizing turns an interference problem into two independent sizing problems.

The full-queue policy menu

Lesson 03's q_push blocks when the queue is full. That's one policy among four, and the choice belongs to whoever operates the pool — so know the menu and each option's bill:

block the submitter    backpressure propagates upstream naturally;
                       submitters stall, and their own callers feel it
reject (submit fails)  pool protects itself; every caller now needs
                       retry/fallback logic it will get wrong unevenly
caller-runs            the submitting thread executes the task itself;
                       self-throttling — but the submitter's latency
                       is hijacked by work it meant to hand off
drop silently          acceptable ONLY for discardable work
                       (sampled telemetry); a bug anywhere else

The interview-grade observation: this is the same lesson the ring buffer taught — a full bounded structure forces a policy, and the policy is a seam. Making it pluggable per-pool (a RejectionPolicy the constructor takes) keeps mechanism and policy apart, exactly like the parking lot's assignment strategy.

Work stealing: when one queue becomes the bottleneck

Lesson 03 ended on the honest ceiling: every submit and every take crosses one mutex. For millisecond tasks, irrelevant; for microsecond tasks, the lock handoff dominates. The scaling shape used by serious runtimes is work stealing: give each worker its own deque, submit to the submitting thread's local deque where possible, and let idle workers steal from the tail of others' deques.

What it buys: the common path (a worker taking its own next task) touches only its own deque — contention collapses. What it costs, stated plainly: global FIFO is gone (tasks interleave across deques), the stealing protocol is genuinely subtle concurrent code, and observability gets harder because there's no longer one queue to measure. The seam observation again: because our pool funnels everything through one queue interface, work stealing is an internal replacement — submitters and the worker-loop contract survive unchanged.

Priorities: cheap mechanism, expensive semantics

Mechanically, priorities are a small edit — the FIFO becomes a priority queue, q_take pops the highest class first. The semantics are where the cost lives, and naming it is the senior move: under sustained load, strict priority means the lowest class never runs — starvation by design. Real systems blunt that with aging (waiting tasks gain priority) or weighted service across classes. "One-line change to the queue, a real policy question about starvation" is the complete answer.

Metrics: a shared pool must be observable

The prompt's backstory — teams spawning ad-hoc threads — means this pool becomes shared infrastructure, and shared infrastructure without gauges is undebuggable. The minimal set, each answering a question an operator will actually ask:

queue depth            are we keeping up?  (the leading indicator)
time-in-queue          what latency is the pool itself adding?
worker utilization     saturated (all busy) or oversized (mostly idle)?
tasks failed           how much user code is throwing? (the handles
                       know — count as they complete with errors)
submit outcomes        how often does the full-queue policy fire?

Queue depth and time-in-queue fall out of the queue for free; utilization is a busy-counter around the task call. A sentence of instrumentation per metric — and offering them unprompted reads as someone who has operated a pool, not just implemented one.

Key takeaway

The pool's variations all land on seams: sizing is a workload argument (cores for CPU-bound, waiting-fraction or a separate pool for IO-bound), the full-queue response is a pluggable policy with four options and four bills, work stealing replaces the queue's internals when the single lock becomes the ceiling, priorities are cheap mechanically and expensive semantically (starvation), and metrics make a shared pool operable. Knowing which seam absorbs which change is the understanding being graded.

Enjoying the preview?

Create a free account to unlock the rest of this course, the in-browser judge, and live AI mock interviews.

Sign up free to continue