Rules and Models Together

In one line: a model cannot be changed in twenty minutes, cannot explain itself to a regulator, and cannot encode a policy that was decided rather than learned — so the rules engine stays.

The instinct to resist

The reflexive answer is that rules are legacy and a model replaces them. It is wrong, and every production fraud system disagrees with it.

Four reasons, and none is nostalgia.

Latency to respond. An attack starts at two in the morning. Retraining, validating and deploying a model takes hours at best. Writing a rule takes minutes. When an attack is live and costing money per second, the ability to intervene now is worth more than the elegance of how.

Explainability. "Declined because this device attempted eleven cards in nine minutes" is a sentence you can put in front of an analyst, a customer or a regulator. "The gradient-boosted model scored 0.87" is not.

Policy that was decided, not learned. Sanctions lists, prohibited jurisdictions, contractual restrictions, a merchant category you do not serve. These are not predictions to be estimated — they are constraints, and a model can only ever approximate a constraint.

Known-bad memory. A card confirmed stolen, a device confirmed fraudulent, an account confirmed compromised. That is a lookup, not an inference, and asking a model to rediscover it from features is strictly worse than remembering it.

The division of labour

The clean split, and it is the same test as the recommendation policy layer arriving from a different angle.

Constraints and memory go in rules. Estimation goes in the model. A rule that says "decline if this device has attempted more than twenty cards in ten minutes" is really a weak model with one feature and a hand-picked threshold — and it belongs in the model as a feature, unless it exists because it can be changed instantly.

That is the useful test to state: if the reason for a rule is speed of change or it is a constraint, keep it as a rule. If the reason is it predicts fraud, it is a feature.

How they combine

Three arrangements, in increasing order of maturity.

Rules as a pre-filter. Hard constraints run first — sanctions, known-bad, blocked jurisdictions — and short-circuit. Nothing reaches the model that was never going to be permitted, which also saves the scoring cost.

Rules as post-model overrides. The model scores, then rules can force a decline regardless — or, importantly, force an approve. Allow-lists matter: a long-standing customer whose behaviour genuinely changed should not be blocked by a model that has correctly noticed something unusual.

Rules as model features. A rule's boolean output becomes an input, so the model learns how much weight to give it in context. This is the most powerful arrangement and the slowest to change, since altering the rule now requires retraining.

Most systems use all three, at different points, for different reasons.

Where rule sets go wrong

The failure mode of every mature fraud system, and naming it unprompted is a strong signal.

Each incident adds a rule. Nobody removes any, because removing one might let fraud through and nobody wants to own that. After three years there are eight hundred rules, they interact in ways nobody can predict, and a large fraction fire on almost nothing while a handful are quietly declining a lot of legitimate traffic.

Four controls:

Every rule has an owner and an expiry. A rule written for a specific attack should expire when that attack stops, and re-justifying it is cheaper than debugging its interactions in year three.

Measure each rule's marginal contribution. How much fraud does it catch that nothing else would have caught, and how many legitimate transactions does it decline? Many rules turn out to be pure false-positive generators once the model improved past them.

Simulate before shipping. Replay recent traffic through the proposed rule and see exactly what it would have blocked. This should be a self-service tool, because a rule that cannot be tested before deployment will be deployed untested during an incident.

Cap the total. A hard limit on rule count forces removal, which is otherwise nobody's job.

The rule that is really a threshold

One specific trap, from the adversarial lesson.

A rule of the form "decline above £500 from accounts under 30 days old" is the most probeable object you can ship. It is a single threshold on two observable fields, discoverable by bisection in an afternoon, and the result is not less fraud but a great deal of fraud at £499 from 31-day-old accounts.

If the intent is genuinely a constraint — a risk appetite limit — keep it and accept that it is public. If the intent is prediction, it belongs in the model where the boundary is expensive to map.

Simulation and replay

The infrastructure that makes all of this operable, and it is worth proposing explicitly.

Keep a replay environment holding recent traffic with its known outcomes. Any proposed change — a new rule, a threshold move, a new model — runs against it and reports what would have changed: fraud caught, legitimate declined, review volume, revenue impact.

Two properties make it valuable. It turns an argument about whether a change is a good idea into a measurement. And it lets rules be written during an incident with some confidence rather than by instinct, which is when the worst rules get shipped.

Next: the labels, and the ones you will never get.