The Review Queue

In one line: human review is a fixed-capacity resource, so the design question is not whether to use it but how to allocate it.

Capacity is the constraint

A review team can process some number of items per day. That number is roughly fixed in the short term — hiring and training take months — and it is far smaller than the volume of uncertain content.

So the queue is always over-subscribed, and every design decision is an allocation decision. What gets reviewed, in what order, by whom, and what happens to everything that does not fit.

That last part is the one designs skip. Whatever the queue cannot reach receives the automated decision by default, so the automated policy for unreviewed content is doing most of the work, and it deserves as much attention as the review process itself.

Priority is not score

The central point, and it follows from the first lesson.

Ordering by model confidence is the intuitive choice and it is wrong. It reviews the cases the model is most sure about, which are the ones least in need of a human.

The right ordering is expected harm prevented:

priority  =  severity  x  predicted reach  x  uncertainty

Severity, because some violations matter more. Reach, because harm scales with exposure. Uncertainty, because a human adds most where the model is least confident — reviewing something scored 0.98 confirms what you already knew.

Two operational additions. Age matters — an item that has waited too long should escalate, or a permanently busy queue starves everything below the top. And some categories bypass priority entirely: legally mandated response times mean certain items go to the front regardless of reach.

Routing

Not one queue but several, because reviewers are not interchangeable.

By language. A reviewer must understand what they are reading, including slang, dialect and cultural reference. This is the hardest routing constraint and it shapes staffing more than anything else.

By policy specialism. Hate speech, self-harm, terrorism and financial fraud each require different training. Specialists are faster and more consistent within their area.

By region. Local context — political situation, current events, which terms are coded — is often decisive, and it does not travel.

By severity tier. The most severe categories need experienced reviewers with support structures, not whoever is next available.

The consequence: capacity is not one number but a matrix, and it can bind in one cell while another sits idle. A backlog in one language is not solved by spare capacity in another, and a design that models review as a single throughput number will be surprised by that.

Reviewer wellbeing is a design input

Not a footnote, and treating it as one is a signal in itself.

Reviewing the most severe categories causes real psychological harm. That is a human matter first, and it is also a systems constraint: turnover is high, training is expensive, and a burnt-out reviewer is less accurate.

Four things the design can do:

Reduce unnecessary exposure. Blur or grayscale by default with click-to-reveal, mute audio, show thumbnails rather than full media. Most decisions do not require seeing the content at full fidelity, and defaulting to reduced fidelity is nearly free.

Cap exposure per shift. Limit consecutive time in severe categories and rotate through lighter queues. This is a scheduling constraint the routing system has to respect.

Use the fast path aggressively. Every item resolved by hash matching is one a person does not see. That is the strongest argument for investing in the known-bad database, beyond speed and cost.

Support the mechanism, not just the policy. Access to help is standard; making it usable — during work hours, without a throughput penalty — is the part that determines whether anyone uses it.

Agreement, and what it tells you

The measurement that makes the queue improvable.

Route a sample of items to multiple reviewers and measure how often they agree. It is cheap and it answers three questions at once.

Where the policy is unclear. Low agreement on a category means the guidelines are underspecified, not that the reviewers are poor. That is a policy fix.

Which reviewers need support. An individual consistently out of step with their peers needs training or a different queue.

What the ceiling is. Agreement caps model accuracy. A model matching reviewers at their own agreement rate is done, and further work belongs on the policy.

Sample deliberately across categories rather than uniformly, since agreement varies enormously — spam is nearly unambiguous, hate speech is not.

Reviewer decisions are your training data

The loop that determines whether the system improves.

Every decision is a labelled example on exactly the distribution the model finds hard, which makes it the most valuable training data available. Two things have to be true for that to work.

Capture the reasoning, not just the verdict. Which policy, which clause, which part of the content. A binary outcome is a much weaker label than a categorised one.

Watch for the loop. The queue is filled by the model, so reviewers see what the model sends them — and training on those decisions reinforces the existing boundary. The fix is the same as everywhere else in this course: route a random sample of unflagged content to review, so the training set includes cases the model was confident about and wrong on.

That random sample is the first thing cut when the queue is busy, and it is the thing that keeps the system from converging on its own beliefs.

Next: the violations that are networks rather than posts.