Serving and Degradation

In one line: the decision is synchronous and the deadline is real, so every dependency is on the critical path and every one of them can fail.

The budget

A card authorisation has a few hundred milliseconds end to end, and fraud scoring gets a slice of it — typically tens of milliseconds. Overrun and the payment network times out, which is worse than any decision you could have made.

Where the time goes, and the ordering is deliberate:

The pattern recurs from every other chapter: feature computation dominates, not the model. A design that budgets carefully for inference and vaguely for feature fetching has the proportions backwards.

The optimisation that follows is also familiar: issue the independent lookups in parallel, and treat sequential per-entity fetches as the thing to eliminate. Card velocity, device velocity, account velocity and graph features do not depend on each other.

Fail open or fail closed

The question with no comfortable answer, and the one an interviewer will press.

The velocity store times out. You have no counts. Do you approve or decline?

Fail closed and a dependency outage becomes a payment outage. Every customer is declined, revenue stops, and the incident is far more expensive than the fraud it prevented. On a consumer platform this is usually the wrong answer, and it is the answer a naive design gives.

Fail open and you approve everything for the duration — which attackers notice quickly, because they are watching. A known fail-open path is an invitation.

Neither is right unconditionally, and the strong answer is that this is a per-segment policy rather than a global switch.

The middle branch is the good answer and it is available precisely because the decision is three-way rather than binary. When you cannot assess risk, you do not have to guess — you can ask. Degrading toward friction rather than toward approve or decline is the move that reads as experience.

Two supporting details: fall back to a simpler model that uses only the signals still available, rather than scoring with nulls the model was never trained on; and log every degraded decision distinctly, so the population is identifiable later and can be excluded from training.

Graceful degradation, in order

The ladder, each rung worse and each safe.

Full pipeline, with every signal. Then a reduced model on whatever signals remain. Then rules only — known-bad lists and hard constraints, which are cheap and always available. Then a static policy by amount and customer tenure. Then the payment processor's own risk decision, if you sit in front of one.

The bottom rung is worth naming: on many platforms the acquirer or issuer performs their own fraud checks, so failing all the way through does not mean zero protection. Knowing what sits below you changes how aggressively you need to fail closed.

Consistency and the double-spend

The distributed-systems problem underneath, and it is the one candidates with a systems background can differentiate on.

Velocity counters are shared state updated by many concurrent requests. If two transactions from the same card arrive simultaneously on different nodes, both may read a count of nine and both may be approved, when the tenth should have been blocked.

The strict fix is a coordinated increment — a lock or a transaction — on every request, which the latency budget forbids.

The workable answers, in order of preference:

Partition by entity. Route all requests for a card to the same node, so its counter is local and consistent. Works well and needs consistent hashing plus a plan for rebalancing.

Accept small overcounting. An atomic increment on a fast store without global ordering is nearly always correct and occasionally lets one extra through. On a threshold of ten, an off-by-one is immaterial.

Reserve then confirm. Increment optimistically before the decision and decrement if declined. More correct, more moving parts.

Same shape as budget pacing in the ads chapter and as distributed rate limiting generally — and saying so is worth doing, because it shows the problem is a familiar one rather than a novel difficulty.

Two paths, not one

The design point that resolves most of the latency tension.

The synchronous path does what must happen before the money moves: known-bad checks, velocity, the model, the decision. Tens of milliseconds, and ruthlessly scoped.

The asynchronous path does everything that can happen after: graph updates, expensive enrichment, secondary models, analyst queueing, and re-evaluation as more evidence arrives.

Splitting them means the expensive analysis does not have to fit the budget. And it pairs with the reversibility point from the three-way decision: where an outcome is reversible for hours, the asynchronous path can still catch it before dispatch, which converts a hard real-time constraint into a soft one.

Logging

Log the decision, the score, every feature value used, which rules fired, and the model version — for every transaction, including approvals.

Three reasons and all of them bite. Debugging a decision months later needs the feature values as they were, because they cannot be recomputed once the windows have moved. Regulatory obligations require demonstrating why a decision was made. And it is the training data — a decision logged without its features is a label with no example attached.

Next: the whole thing, as an interview.