Skip to content

From your data to evidence

In Your first decision you typed the evidence numbers in by hand. In a real service they come from your own records: the accounts, patients or posts related to the case in front of you, and what happened to them. This tutorial takes one small set of related cases, draws it, turns it into evidence with one function call, and shows why the library acts on it. Then it removes most of the links and shows why the same decision is held.

It takes about 15 minutes. You need the package installed (Install) and should have read Your first decision. The example is a card fraud desk, but every step has a counterpart in other domains; the table in step 6 lists some.

The situation

A card payment from account acct-7731 is flagged suspicious by the fraud model. The desk's policy freezes suspicious cards:

from grounded_decision import parse

policy = parse("test(clean) ; act(pass) | test(suspicious) ; act(block)")

Before freezing, the desk wants to know what the related cases say. Here the related cases are the 16 accounts reachable from acct-7731 through shared devices in the last 30 days, and each one has the outcome of its last fraud review: confirmed fraud or cleared. acct-7731 itself is left out, because it has no review outcome yet. (The service in Build a decision service makes the other choice: it includes the flagged account and counts never-reviewed accounts as cleared. Either is reasonable; make the same choice every time.)

Four devices were shared. Two accounts are linked when they logged in from the same device. Fourteen of the 16 accounts were confirmed as fraud (red); two were cleared (green).

flowchart LR
    classDef fraud fill:#f6c4c4,stroke:#b03030,color:#000
    classDef cleared fill:#c8ecc8,stroke:#2f7d32,color:#000
    classDef device fill:#e8e8e8,stroke:#666,color:#000
    A(["phone_A"]):::device
    B(["phone_B"]):::device
    C(["laptop_C"]):::device
    D(["phone_D"]):::device
    a0((0)):::fraud --- A
    a1((1)):::fraud --- A
    a2((2)):::fraud --- A
    a3((3)):::fraud --- A
    a4((4)):::fraud --- A
    a5((5)):::fraud --- A
    a5 --- B
    a6((6)):::fraud --- B
    a7((7)):::fraud --- B
    a8((8)):::fraud --- B
    a9((9)):::fraud --- B
    a9 --- C
    a10((10)):::fraud --- C
    a11((11)):::fraud --- C
    a12((12)):::fraud --- C
    a13((13)):::fraud --- C
    a12 --- D
    a13 --- D
    a14((14)):::cleared --- D
    a15((15)):::cleared --- D

Account 5 used both phone_A and phone_B; account 9 used phone_B and laptop_C; accounts 12 and 13 used laptop_C and phone_D. Those overlaps tie the four groups into one ring. The only disagreement is on phone_D, where two confirmed accounts share a phone with two cleared ones.

2. Turn the drawing into lists

The library wants two lists: one outcome per account, and the links as pairs of account numbers. A device shared by \(k\) accounts links every pair of them.

from itertools import combinations

# Which related accounts logged in from each device (last 30 days).
logins = {
    "phone_A": [0, 1, 2, 3, 4, 5],
    "phone_B": [5, 6, 7, 8, 9],
    "laptop_C": [9, 10, 11, 12, 13],
    "phone_D": [12, 13, 14, 15],
}
# Last fraud review per account: 1 = confirmed fraud, 0 = cleared.
outcomes = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]

# Two accounts are linked if they used the same device. Each pair once.
edges = sorted(
    {pair for accounts in logins.values() for pair in combinations(accounts, 2)}
)
print(len(edges), "links")

40 links

phone_A gives 15 pairs, phone_B 10, laptop_C 10 and phone_D 6. Accounts 12 and 13 share two devices but count as one link, so the total is 40.

3. Compute the evidence

snapshot_from_graph counts what it needs from the two lists. You add three things only you know: your decision line for link density, the calibration set, and this case's unusualness score (both explained in step 5).

from grounded_decision import snapshot_from_graph

# Unusualness of 30 recent flags the review confirmed: 1 - model confidence.
calibration = [round(0.02 + 0.01 * i, 2) for i in range(30)]

evidence = snapshot_from_graph(
    outcomes,
    edges,
    p_threshold=0.10,  # the desk calls a group a ring above 10% density
    scores=calibration,
    label_score=0.09,  # the model was 91% confident on acct-7731
    observations={"flagged_account": "acct-7731", "window_days": 30},
)
print(
    f"n={evidence.n} p_hat={evidence.p_hat:.3f} "
    f"degree={evidence.degree:.3f} lambda_hat={evidence.lambda_hat:.3f}"
)

n=16 p_hat=0.333 degree=5.000 lambda_hat=0.800

Each number, in the desk's terms:

  • n = 16: there are 16 related accounts.
  • p_hat = 0.333 (link density): of the 120 possible pairs among 16 accounts, 40 are linked, so a third of all pairs share a device.
  • degree = 5.0 (average links per account): 40 links, each touching two accounts, spread over 16 accounts: \(2 \times 40 / 16\).
  • lambda_hat = 0.8 (neighbour agreement): 36 of the 40 links join accounts with the same outcome. Only the four links between {12, 13} and {14, 15} disagree. Agreement on 90% of links, where a coin flip would give 50%, is turned into a number from 0 (no better than chance) to 1 (always the same): \((2 \times 0.9 - 1)/(2 - 1) = 0.8\).

4. Decide

from grounded_decision import Event, Module

module = Module.from_expr("card_review", policy)
record = module.decide(Event("suspicious", {"account": "acct-7731"}), evidence)
print(record.verdict.value, record.action, record.proposed_action)
for channel in record.channels:
    print(
        f"{channel.name.value:15} {'PASS' if channel.passed else 'FAIL'}  {channel.summary}"
    )

act block block
certificate     PASS  star-height 0 ≤ 3; 3 states ≤ 256
game            PASS  parity_win=True nonnegative_mean=True
reconstruction  PASS  d λ² = 3.2000 vs 1+0.050 (supercritical)
threshold       PASS  evidence density outside operational window (theorem width=1.1e+05, vacuous=True)
conformal       PASS  conformal p=0.774 vs α=0.100 (cover)
support         PASS  support not measured

The card is blocked. Three of the checks that read the evidence pass for these reasons; the fourth, support, is not measured yet (the last section of this page turns it on):

  • Do the related cases agree? (reconstruction) The evidence strength is average links × agreement² = \(5 \times 0.8^2 = 3.2\). Above 1, linked accounts carry real information about each other; the library asks for more than 1.05. 3.2 passes comfortably.
  • Are we on a knife-edge? (threshold) With 16 accounts, the library treats any density within 0.209 of your line as too close to call: the window runs from 0 to 0.309 around the line at 0.10. The density 0.333 is just outside it. (The theorem width in the summary is the width the underlying theorem would give at this size; it is far wider than 0 to 1, which is what vacuous=True means, so the practical window is used instead. See Trust levels and honest limits.)
  • Have we seen cases like this? (conformal) 23 of the 30 calibration scores are at least as high as 0.09, so the p-value is \((1 + 23)/(1 + 30) = 0.774\), well above the 0.10 bar. This flag looks like the flags the review usually confirms.

The first two checks look only at the policy, which is unchanged from Your first decision.

5. The same accounts, thinned out

Now suppose the desk only keeps 7 days of login history. The same 16 accounts, with the same review outcomes, share far fewer devices:

recent_logins = {
    "phone_A": [0, 1, 2],
    "phone_B": [6, 7],
    "laptop_C": [9, 10, 14],
    "phone_D": [13, 15],
}
recent_edges = sorted(
    {pair for accounts in recent_logins.values() for pair in combinations(accounts, 2)}
)
thin = snapshot_from_graph(
    outcomes, recent_edges, p_threshold=0.10, scores=calibration, label_score=0.09
)
print(len(recent_edges), "links")
print(
    f"n={thin.n} p_hat={thin.p_hat:.3f} degree={thin.degree:.3f} lambda_hat={thin.lambda_hat:.3f}"
)

module.reset()  # a new case starts at the beginning of the policy
held = module.decide(Event("suspicious", {"account": "acct-7731"}), thin)
print(held.verdict.value, held.action, held.proposed_action)
print(held.reason)

8 links
n=16 p_hat=0.067 degree=1.000 lambda_hat=0.250
abstain None block
d λ² = 0.0625 vs 1+0.050 (at or below KS; abstain); evidence density inside operational window (theorem width=1.1e+05, vacuous=True)

Nothing is carried out, block stays on the record for an analyst, and the reason names two failed checks:

  • Agreement collapsed. Only 8 links remain, one per account on average, and three of them (9–14, 10–14, 13–15) join a confirmed account to a cleared one. 5 of 8 agree, so agreement is \(2 \times 0.625 - 1 = 0.25\) and strength is \(1 \times 0.25^2 = 0.0625\), far below 1.05. ("KS" in the reason is Kesten–Stigum, the name of the line at 1.)
  • The density sits on the knife-edge. 8 of 120 pairs is 0.067, only 0.033 below the line at 0.10 and well inside the 0.209 window. A few more shared logins tomorrow would push it over the line.

The unusual-case check still passes; the model's confidence did not change. One passing check is not enough.

The lesson for your own data: what counts as a link, and how far back you look, decide what the checks see. The same accounts with a shorter history are, honestly, less evidence.

6. What each evidence field means

Field Fraud desk Clinical triage Content moderation Read by Where the value comes from
n related accounts similar past visits posts matching this one knife-edge measured from your graph
p_hat share of account pairs that share a device share of visit pairs judged similar share of post pairs that near-match knife-edge measured from your graph
p_threshold density at which the desk calls a group a ring density at which visits count as one pattern match density that triggers removal knife-edge your domain, or learned
degree links per account (accounts it shared a device with) similar visits per visit matches per post agreement measured from your graph
lambda_hat how often linked accounts share a review outcome, beyond chance how often similar visits share an outcome how often matching posts got the same moderation agreement measured from your graph
n_states number of possible outcomes (2: fraud, cleared) 2: deteriorated, did not 2: removed, kept agreement your label definition
scores unusualness of recent confirmed flags unusualness of recent correct triage calls unusualness of recent upheld removals unusual case learned from history
label_score this flag's unusualness this patient's unusualness this post's unusualness unusual case your model, scored as in history
support share of related accounts whose outcome matches the action (block: fraud; pass: cleared) share of similar visits with the outcome the action implies share of matching posts with the moderation the action implies support measured from your graph, given your action-to-outcome mapping
observations your notes (flagged_account, window) ward, shift classifier version nothing you

The last column is explained on Where every number comes from: "measured" values are computed from your graph on every decision, "learned" values come from your labelled history with a certified error rate (Learn parameters), and the rest are definitions you make.

Points that trip people up:

  • lambda_hat is not a model probability. It is measured from the links and outcomes. A strongly negative value (linked cases tend to have different outcomes) also counts as informative, because the check squares it.
  • When one label dominates everywhere, consider agreement_baseline="label_mix". If 90% of all accounts are clean and your links may be unrelated to outcomes, random links agree most of the time and the default measure reads that as signal. With 60 accounts, 10% fraud and 146 random links, the default gives lambda_hat 0.493 and evidence strength 1.18 (passes); snapshot_from_graph(..., agreement_baseline="label_mix") gives −0.145 and 0.10 (holds). Do not switch for rings like the one on this page: label_mix compares with the graph's own label mix, so a ring that is nearly all fraud has little variation inside it and scores near zero, although it is strong evidence. The default (uniform) is the measure the result behind this check assumes. Which way the related cases point is the support check's job (last section).
  • The unusualness score: higher means more unusual. Here it is 1 − the model's confidence, so a hesitant flag scores high. Compute it the same way for the calibration set and for the new case. The calibration set is best learned from labelled history (Learn parameters); Choose thresholds and calibration covers other choices and how many calibration scores you need.
  • An empty calibration set passes. With no scores, the unusual-case check cannot fail unless you set Limits(require_conformal_calibration=True). Turn it on in production.
  • observations is for you. No check reads it, and it is not copied into the decision record. Store evidence.to_dict() next to the record if you want it in your audit log.
  • n_states: use the number of outcome values in outcomes. snapshot_from_graph uses it only to turn agreement into lambda_hat.

7. Small groups are held most of the time

The knife-edge window depends only on how many related cases there are. Half-widths from the library's operational_width at its default settings:

Related cases (n) 6 10 16 30 40 80 200 1000
Window, either side of line 0.5 0.304 0.209 0.139 0.118 0.084 0.057 0.034

With 6 or fewer related cases the window covers almost every density, so the knife-edge check fails unless the density is more than 0.5 from your line. Under about 10 cases it fails unless the density is 0.3 or more away. That is deliberate (a few links either way change the picture), but if most of your cases have small neighbourhoods, most of your decisions will be held. Plan the review queue for it, or gather more related cases per decision.

Which way does the evidence point?

Neighbour agreement, the knife-edge check and the unusual-case check ask whether the related cases are informative enough to act on. They do not ask which way they point. Send clean for the same account, with the same dense fraud ring as evidence:

module.reset()
cleared = module.decide(Event("clean"), evidence)
print(cleared.verdict.value, cleared.action, cleared.reason)

act pass all channels passed

The pass goes through, because the evidence is strong and the support check had nothing to measure. Turn it on by telling snapshot_from_graph which outcome the proposed action needs among the related accounts (supporting_label). peek_proposed_action gives the action the policy would propose, without deciding:

# Which outcome each action needs among the related accounts.
SUPPORTING_LABEL = {"block": 1, "pass": 0}  # 1 = confirmed fraud, 0 = cleared

for event_name in ("suspicious", "clean"):
    module.reset()
    proposed = module.peek_proposed_action(event_name)
    directed = snapshot_from_graph(
        outcomes,
        edges,
        p_threshold=0.10,
        scores=calibration,
        label_score=0.09,
        supporting_label=SUPPORTING_LABEL[proposed],
    )
    decision = module.decide(Event(event_name), directed)
    print(
        event_name,
        proposed,
        f"support={directed.support:.3f}",
        decision.verdict.value,
        decision.reason,
    )

suspicious block support=0.875 act all channels passed
clean pass support=0.125 abstain support 0.12 vs minimum 0.50 (does not support the proposed action)

14 of the 16 related accounts are confirmed fraud: a share of 0.875 supports block, and a share of 2/16 = 0.125 does not support pass, so the pass is held for an analyst. The default minimum is 0.5 (Limits.min_support); learning can choose it from your history.

The support check measures only what you tell it: the action-to-outcome mapping is yours, and so are the links and outcomes. If they are wrong, so are the numbers. More: Do the related cases support this action?

Next