Skip to content

Naive vs grounded

Most automated decisions follow a simple rule: the model or monitor says "suspicious", so block; it says "low risk", so discharge. That rule acts every time, including the times when nothing but the score supports it. This page shows, across three real-world cases, the one pattern this library adds: before acting, check that the related cases and recent history are strong enough to support an automatic action, and hand the case to a person when they are not.

It is written for anyone deciding whether this approach fits their problem. No maths is needed; numbers are in collapsed boxes, and every one comes from running the example named next to it.

The pattern

Today's rule (the naive policy in the examples) has one step:

something happened → carry out the action the rule names.

The grounded policy keeps that step but treats its result as a proposed action, and adds a second question before carrying it out:

is the evidence around this case strong enough to act on without a person?

flowchart LR
    E[Event: suspicious, low_risk, …] --> P[Policy proposes an action]
    R[Related cases and recent history] --> C{Six checks}
    P --> C
    C -- all pass --> A[Act: carry it out]
    C -- any fails --> H[Hold: record the proposal and the reason, send to a person]
    C -- policy names a safer step --> D[Degrade: carry out the reduced action]

"Evidence" here means the related cases: the other accounts that shared a device, the past patients with similar vital signs, the borrowers at the same employer. Four of the six checks read them:

  • Do the related cases agree? Are there enough links between them, with outcomes consistent enough, that they say something about this case? (details)
  • Are we on a knife-edge? Is the pattern of links far enough from your cut-off that a few missing records would not change the reading? (details)
  • Have we seen cases like this? Does this case's score look like the scores of recent cases that were decided correctly? (details)
  • Do the related cases support this action? Do most of them carry the outcome the proposed action implies? It runs only where the example measures it, here fraud review and clinical triage. (details)

The other two check the policy itself when it is built (it is not nested too deeply, and the workflow can keep making progress); they pass in every example on this page. The six checks overview covers all six.

The proposed action is always recorded, even when it is held, so a reviewer sees what the rule wanted and why it was stopped.

Three contrasts

Clinical triage: a "low risk" patient with too little history

Example: uv run grounded-decision examples clinical_triage. Full page: Clinical triage.

An emergency department scores three patients. The rule discharges low risk and escalates high risk.

Patient Related past visits Today's rule Grounded
Low risk, heart rate 72, oxygen 98 % 17 closely similar visits, none deteriorated discharge act: discharge
High risk, heart rate 128, oxygen 90 % 14 similar visits, 12 deteriorated escalate act: escalate
Low risk, heart rate 88, oxygen 95 % only 5 loosely similar visits, outcomes mixed discharge hold: keep in intake, page the charge nurse

The third patient's score is fine. What is missing is history: five past visits whose outcomes are no better than a coin flip at predicting each other, too few to say anything. The grounded policy keeps the patient in intake and gives the nurse the proposed discharge and the reason.

The numbers (third patient)

Links per visit \(d = 1.6\); agreement \(\lambda = 0\); evidence strength \(d\lambda^2 = 0\), below the 1.05 line, so the agreement check fails. With 5 visits, the knife-edge window is at its widest (half-width 0.5) around the 0.15 cut-off, and the measured density 0.40 is inside it, so that check fails too. The unusual-cases check passes (p = 1.0): the demo sends a fixed score, so this check cannot fail here.

Fraud review: a block that rests on one shared laptop

Example: uv run grounded-decision examples fraud_review. Full page: Fraud review.

Two $840 authorisations flagged suspicious by the risk model, and a $12 grocery purchase the model calls clean.

Card Related accounts Today's rule Grounded
A 16 accounts sharing 8 phones in rotation, 13 confirmed fraud block act: block
B 4 accounts in a chain of 3 shared laptops, outcomes inconsistent block hold for a fraud reviewer
C (clean) the same ring as card A pass hold for a fraud reviewer

Card B is the classic false freeze: a family laptop or an internet café links the customer to a past fraud case. The grounded policy does not block; it sends the case to review, and your authorisation path applies its fallback (for example, step-up authentication) meanwhile. Card C is the opposite mistake: the model says clean, but 13 of the 16 linked accounts were confirmed fraud, so the grounded policy does not pass it automatically either.

The numbers

Card A: \(d = 5.0\), \(\lambda = 0.55\), \(d\lambda^2 = 1.51\), above 1.05; support for block 13/16 = 0.81. Card B: \(d = 1.5\), \(\lambda = 0.33\), \(d\lambda^2 = 0.17\), below; and with only 4 accounts the knife-edge check fails as well. Card C: the same strong evidence as card A, but support for pass is 3/16 = 0.19, below the 0.5 minimum.

Credit underwriting: under the cut-off, but unlike anyone who repaid

Example: uv run grounded-decision examples credit_underwriting. Full page: Credit underwriting.

The scorecard approves anyone under a 12 % predicted default risk.

Applicant Today's rule Grounded Why
FICO 760, 2 % risk approve act: approve Looks like recent borrowers who repaid.
FICO 610, 11 % risk approve hold for an underwriter Under the cut-off, but riskier than every one of the last 40 approved borrowers who repaid.
FICO 480, 45 % risk deny act: deny Peers at the same closed employer mostly defaulted.

Here the related cases are fine; the hold comes from the unusual-cases check. The lender's recent history contains no repaid loan scored as risky as this applicant, so an automatic approval would be a step outside what the lender has seen work. In effect, this check sets a stricter cut-off just inside the policy's own, wherever recent history runs out.

The numbers (second applicant)

Unusualness p-value \(= (1 + 0)/(1 + 40) = 0.024\): none of the 40 calibration scores is at least 0.11. That is below α = 0.10, so the check fails. The related borrowers are strong evidence (\(d\lambda^2 = 5.45\)) and pass.

What the three have in common

  • The rule's direction is unchanged. The grounded policy never turns a discharge into an escalation or a block into a pass. It either does what the rule proposed, or holds it (including when the related cases point the other way).
  • Holds are specific. Each hold names the check that failed and why, in the record's reason, so the person picking it up knows what is missing: more related cases, a clearer pattern, or a case outside recent experience.
  • Acts are as fast as before. When the evidence is strong, the action goes through automatically; the library only stops the cases the rule could not justify.

What this does not do

Direction is checked only where support is measured

The agreement check asks whether related cases agree with each other, not whether they agree with the proposed action. Only the support check asks that, and only when the evidence carries the support share, as fraud review and clinical triage do. In the other examples on this site, a clean authorisation on a card inside a fraud ring would act pass, because the ring is strong evidence for whatever is proposed. See Fraud review.

  • It does not make your model more accurate. It decides when the model's call can be carried out without a person.
  • A hold is not free. Someone must pick it up within a time limit, and your system needs a fallback for when they don't (for example, step-up authentication for a held card, or keeping a patient in intake). Each case page has a "What happens next" section.
  • With very few related cases (under about ten), the knife-edge check holds almost everything. If your decisions usually have only a handful of related cases, expect many holds or tune the cut-off: Choose thresholds and calibration.

When this is not worth it

  • Every decision is cheap to reverse and nobody reviews them anyway.
  • You have no outcomes for related cases (no confirmed fraud labels, no follow-up on discharged patients). The checks have nothing to read.
  • There is nobody to take a hold. A hold with no owner is a silent default; decide the fallback first.

Where to go next