Skip to content

The six checks

Before the library carries out an action your policy proposes, it asks six yes/no questions about the policy and the evidence. If any answer is "no", it holds the decision and writes down which question failed and why. This page puts the six questions side by side, explains why all six must pass, and shows how the last of them, the support check, catches evidence that is strong but points the other way.

It assumes you know the words from Anatomy of a decision: a policy turns an event into a proposed action, and the evidence describes the related cases around this decision. Each check has its own page with a worked example; this page is the map.

The six questions at a glance

Check (name in records) Plain question Looks at When it runs Typical reason it says "no" Main setting
Policy shape (certificate) Is the policy simple enough to read and audit? The policy text Once, when the module is built Repetition nested more than three deep; too many states max_star_height (3), max_dfa_states (256)
Progress (game) From its start, can the workflow keep making progress? Priorities, payoffs and owners of states Once, when the module is built A loop that never proposes an action; every cycle costs Priorities and payoffs on your states
Neighbour agreement (reconstruction) Do the related cases agree strongly enough to tell us anything? Links per case and how often linked cases agree Every decision Too few links, or links whose labels disagree reconstruction_margin (0.05)
Knife-edge (threshold) Is the link density too close to our cut-off to trust? Number of cases, link density, your cut-off Every decision Density near the cut-off, or a small graph p_threshold (yours), operational_fk_prefactor (2.0)
Unusual cases (conformal) Is this case's score like the scores of cases we were right about? This case's unusualness score, the calibration set Every decision Score higher than nearly every calibration score conformal_alpha (0.10), require_conformal_calibration
Support (support) Do the related cases point the same way as the proposed action? Share of related cases carrying the label the action implies Every decision Most related cases carry the opposite label min_support (0.5), require_support

The names in brackets are the values of the name field on each check's entry in the decision record (the code calls an entry a "channel"). The defaults are fields of Limits; the full list is in Defaults and limits. The defaults of the four data checks are only starting points: their settings (and the calibration set) are learned from your labelled history with a certified error rate (Learn parameters from your history). The structure checks' settings are engineering budgets and parts of your policy, not learned (Where every number comes from).

flowchart LR
    subgraph build["Once, when the module is built"]
        P[Policy text or chart] --> C1[Policy shape]
        P --> C2[Progress]
    end
    subgraph each["Every decision"]
        E[Event] --> S[Policy step] --> PA[Proposed action]
        EV[Evidence] --> C3[Neighbour agreement]
        EV --> C4[Knife-edge]
        EV --> C5[Unusual cases]
        EV --> C6[Support]
    end
    C1 --> AND{All six pass?}
    C2 --> AND
    C3 --> AND
    C4 --> AND
    C5 --> AND
    C6 --> AND
    PA --> AND
    AND -- yes --> ACT[act: carry out the proposed action]
    AND -- no --> HOLD[abstain: hold, keep the proposal on the record]

The two structure checks (policy shape and progress) are worked out once when you build the module, and their answers are copied into every record. The four data checks are recomputed from the evidence you pass with each event.

Why all six must pass

Each check looks at a different way a decision can be unsupported. The examples below make each one fail on its own while the others pass, so you can see that none of them is redundant:

Example (run with uv run grounded-decision examples <id>) Situation The only check that fails
step_star_height_reject A policy with four nested loops Policy shape (at build time; no decision is made)
step_reconstruction_gate Carol's team: 20 people, 60 links, but who holds the access is scattered Neighbour agreement (\(d\lambda^2 = 0.027\), needs more than 1.05)
step_threshold_window Erin's team: link density 0.21 against a cut-off of 0.20, 50 people Knife-edge (0.01 from the cut-off; the window is ±0.105)
step_conformal_outlier A 3am request from a new country scores 1.5; every past legitimate one scored 0.40–1.18 Unusual cases (\(p = 0.024\), needs more than 0.10)
fraud_review (tick 2) The model says clean, but 13 of the 16 accounts in the card's device ring are confirmed fraud Support (share 0.19, needs at least 0.50)

No shipped example fails the progress check; the progress page shows small charts and policies that do.

The checks are separate, but they are not independent. Neighbour agreement and the knife-edge check read the same graph of related cases, so a thin graph usually fails both. In fraud_review, the flagged card linked to only four accounts fails both at once:

reconstruction   FAIL   d λ² = 0.1667 vs 1+0.050 (at or below KS; abstain)
threshold        FAIL   evidence density inside operational window (theorem width=4.39e+05, vacuous=True)

That is expected: a thin graph is both uninformative and too small to measure a density reliably.

Strong evidence is not the same as supporting evidence

Neighbour agreement asks whether the related cases are informative, not which outcome they point to. The knife-edge check asks whether the density is close to the cut-off, not which side it is on. The unusual-cases check asks whether the score looks familiar, not what it predicts. Only the support check asks about direction.

fraud_review shows why that matters. The same dense ring of 16 linked accounts (\(d\lambda^2 = 1.513\), density 0.333, 13 accounts confirmed fraud) is attached to two events:

Event Proposed action First five checks Support share Verdict
suspicious block pass 0.8125 (fraud) act
clean pass pass 0.1875 (cleared) abstain

The first five checks pass for both, because the evidence is equally strong for either. The support check holds the pass: the related accounts say fraud, the model says clean, and a person should look.

The support check runs only when you measure it: you tell the library which label each action implies (in the example, block implies confirmed fraud and pass implies cleared) and the evidence carries the share. When you do not, the check passes with support not measured, and the direction of the decision again comes entirely from the event your code sends. See Support.

What a check looks like in a record

Every record carries all six checks, passed or not. Here are the four data checks from the held discharge in clinical_triage (tick 2, from uv run grounded-decision examples clinical_triage --json), trimmed to the fields that matter:

{"name": "reconstruction", "passed": false, "trust": "paper",
 "metrics": {"degree": 1.6, "lambda_hat": 0.0, "d_lambda_sq": 0.0, "margin": 0.05}}
{"name": "threshold", "passed": false, "trust": "operational",
 "metrics": {"n": 5, "p_hat": 0.4, "p_threshold": 0.15,
             "operational_width": 1.2426698691192237, "in_operational_window": true}}
{"name": "conformal", "passed": true, "trust": "operational",
 "metrics": {"p_value": 1.0, "alpha": 0.1, "n_cal": 40, "label_score": 0.12}}
{"name": "support", "passed": true, "trust": "operational",
 "metrics": {"min_support": 0.5, "require_support": false, "measured": true, "support": 0.8}}
  • passed is the yes/no answer.
  • trust says how much of a mathematical result the answer rests on: proven, published, or practical. See Trust levels and honest limits.
  • metrics are the numbers the answer was computed from. Each check's page lists its fields.
  • A one-line summary (not shown) is what appears in the record's reason when the check fails.

The full record format is in The decision record.

Where next