Skip to content

Why a score isn't enough

A model score is one number. It can be high because many similar cases clearly point the same way, or because the model is extrapolating from almost nothing, and the number looks the same in both cases. This page explains, without any mathematics, what the library looks at besides the score, and why it holds an action when that extra information is missing.

It assumes you know what a model score or a business rule is. It doesn't assume any code. When you finish, you should be able to explain to a colleague why a discharge, a card freeze and a post take-down can all go wrong in the same way, and what each of the six checks asks.

Three rooms

A hospital at 2 a.m.

A patient arrives. Heart rate 88, oxygen 95%. The triage model tags them low_risk. The department's rule fits on a whiteboard: low-risk patients go home with advice; high-risk patients go to a monitored bed. The tag says low_risk, so the dashboard says "discharge".

What the tag doesn't tell you is how much history stands behind it. If the department has seen 17 visits very like this one, all closely similar to each other, and none of those patients got worse, the tag is standing on solid ground. If it has seen only 5 loosely similar visits, and one of them deteriorated, the tag is a guess. The rule discharges both patients. A wrong discharge is the most expensive mistake an emergency department makes.

A card fraud desk

A card authorisation for an $840 electronics purchase is flagged suspicious. The rule: freeze suspicious cards. Behind the scenes there is a graph: accounts that logged in from the same phone or laptop are linked, and each account carries the outcome of its last fraud review.

Sometimes the flagged account sits in a dense ring: sixteen accounts sharing eight phones, most of them already confirmed as fraud. Freezing is right. Sometimes it sits in a chain of four accounts that once shared a family laptop, with mixed review outcomes. Freezing that card punishes a customer for sharing a computer. The model's score can be equally high in both cases.

A moderation queue

A post is flagged as a likely scam video. The rule removes a post when it closely matches enough posts moderators already removed: when the density of matches is above 30%. This post matches a loose cluster of 30 template posts at a density of 37%. That is just above the line. A few more or fewer matches tomorrow would flip the answer, and the same kind of content would be removed one day and left up the next. Moderators experience that as inconsistency; users experience it as unfairness.

What the three rooms share

Each rule acts on one number: a risk tag, a fraud score, a match density. Each room also has something the number left out: how many similar cases there are and whether they agree, whether the case is like anything seen before, and how close the number sits to the line. This library looks at that missing information before it lets the action go ahead.

Proposal versus permission

The library separates two questions that a score rule answers at once:

  1. What would we do? Your policy answers this. It is a short list of which events lead to which actions: "low_risk proposes discharge; high_risk proposes escalate." The answer is the proposed action.
  2. Are we allowed to do it now? Five checks answer this, using the policy and the evidence: numbers that describe the related cases around this decision.

If every check passes, the verdict is act and the proposed action is carried out. If any check fails, the verdict is abstain: nothing is carried out, the proposal stays on the record, and a person decides. A policy can also name a weaker fallback in advance, such as "serve read-only"; reaching it gives the verdict degrade.

The record is the point. A held discharge is not a lost decision: the clinician who picks it up sees "proposed: discharge; held because only 5 similar visits, and they disagree". Act, abstain, degrade covers the verdicts and how to route them.

The six checks as six questions

Each check asks one plain question and reduces the answer to one number. The first two look at the policy and are worked out once, when the policy is loaded. The last four look at the evidence and run on every decision.

1. Is the policy simple enough to analyse?

A policy can repeat steps ("submit, then approve or reject, and start again"). Repeats can be nested inside repeats. The library counts the deepest nesting, called star height, and refuses policies nested more than 3 deep. A typical approval workflow has star height 1. A policy with four nested repeats is rejected when you load it (star-height 4 exceeds max 3), long before any decision. More: Is the policy well-formed?

2. Can the workflow keep going?

A pager policy that can only ever move into "paging" and stay there, losing a point of value at every step, will never recover. The library turns the policy into a small game and asks whether it can keep running without getting trapped, with an average long-run value of at least zero. For a hand-drawn pager chart whose only states each score −1, the check fails and every decision from it is held.

For a policy in the short text form, "keep going" means "keep reaching a step that proposes an action, or a step that finishes". A policy such as (test(ping))*, which reads ping events forever and never proposes anything, fails. For a hand-drawn chart you say which states the workflow must not circle in forever, and which states someone else (a customer, a pager) controls; the check assumes the worst from those. In the states your service controls, it assumes the service sends the event that keeps the workflow moving, so a pass means progress is possible, not that it will happen. More: Can the workflow keep going?

Draw each related case as a dot, and draw a line between two cases that share something: a device, a symptom pattern, a near-identical image. Two numbers matter. How many lines each case has on average (call it \(d\)), and how much more often linked cases share an outcome than chance would give (the neighbour agreement, \(\lambda\), from 0 for "no better than a coin flip" to 1 for "always the same"). The evidence strength is \(d \lambda^2\). Above about 1, the related cases genuinely tell you something about this one; the library requires more than 1.05.

In the triage demo, the confident patient's 17 similar visits have \(d = 11.06\) links each and full agreement (\(\lambda = 1\)): strength 11.06, pass. The uncertain patient's 5 visits have \(d = 1.6\) links each and agreement no better than chance (\(\lambda = 0\)): strength 0, hold. More: Do the related cases agree?

4. Are we on a knife-edge?

You decide where the line is: the density of links above which a pattern counts (for moderation, 30% of possible matches). Close to that line, a little noise flips the answer. The library computes a window around your line whose width depends on how many cases you have: wide for a few cases, narrow for many. If the observed density is inside the window, it holds.

The moderation post above has 30 related posts, a density of 0.37 and a line at 0.30. With 30 cases the window reaches 0.139 either side of the line (0.161 to 0.439), so 0.37 is inside it: hold. A re-upload of a video moderators already removed 20 times has density 1.0, far outside: act. More: Are we on a knife-edge?

Small groups of related cases

The window is wide when there are few cases. With 16 cases it reaches about 0.21 either side of the line; with 6 or fewer it reaches 0.5, so a density has to be very far from the line to pass. Small neighbourhoods are held by this check most of the time. That is intended (a few links can't establish a pattern), but it surprises people. From your data to evidence shows it with real numbers.

5. Have we seen cases like this?

You give each case an unusualness score (for a lender, the predicted default risk), and you keep the scores of recent cases where the answer turned out right, the calibration set. If this case is more unusual than nearly all of them, the library holds: it has no track record for cases like this. The number is a p-value: the share of calibration cases at least as unusual as this one (counting this one). The default bar is 0.10.

In the credit example, an applicant with an 11% predicted default risk is under the lender's 12% cut-off, so a score rule approves. But they are riskier than every one of the 40 borrowers who repaid last quarter, so the p-value is 1/41 = 0.024, below 0.10: hold. More: Have we seen cases like this?

Strong evidence can point the wrong way. A card flagged clean by the fraud model may belong to an account in a ring of 16 accounts sharing phones, 13 of them confirmed fraud. Checks 3 to 5 pass: the ring is large and consistent. This check counts the related cases that carry the outcome the proposed action implies (pass implies "cleared"): 3 of 16, a share of 0.19, below the default minimum of 0.5: hold. The same ring supports block with a share of 13/16 = 0.81. You tell the library which outcome each action implies; if you don't, the check passes with "support not measured". More: Do the related cases support this action?

What the checks don't do

  • They don't choose the direction. The event your model or rule sends chooses the action. Check 6 can hold an action the related cases contradict, but only if you tell it which outcome each action implies; without that mapping, a clean card inside a dense fraud ring is passed.
  • They don't replace your model. The model still produces the event (suspicious, low_risk) and often the unusualness score.
  • They can't see data you don't give them. If the calibration set is empty, the unusual-case check passes by default unless you turn on require_conformal_calibration. If your graph is wrong, the checks measure the wrong graph.
  • They don't decide what a held case means. Abstain means "this module will not act". A person can still discharge the patient.

Trust levels and honest limits goes further.

Where the math lives

Each check stands on a published mathematical result, and some on machine-checked proofs. You don't need them to use the library. When you want them: checks 1 and 2 use star height and parity and mean-payoff games; check 3 uses the Kesten–Stigum reconstruction bound; check 4 adapts the Friedgut–Kalai sharp-threshold theorem; check 5 is split conformal prediction. The Theory overview states each one, says how much of it the code relies on, and where the code's use departs from the theorem.

Next