Parameter learning: let your own outcomes choose the settings¶

Before a fraud team can let the library block or pass cards on its own, someone has to say how strong the evidence must be, how close to the "this is a ring" cut-off is too close to call, and how unusual a model score may be before a person looks. Nobody knows those numbers up front, and numbers typed in by hand have no provenance: nobody can say what error rate they buy. This page shows how the library learns every one of them from past decisions whose outcome you now know, starting from the one number the business can supply: "at most 5% of automatic decisions may be wrong". It is written for a fraud lead and the data scientist who supports them; the statistics are in collapsed boxes and linked pages.

At a glance


Decision	The same as Fraud review: block the card (`block`) or let the authorisation through (`pass`). What is new is where the check settings come from.
Owner	Fraud operations sets the acceptable error rate; fraud data science runs the learning.
Today's rule	Hand-set settings: evidence strength above 1.05, a knife-edge window with prefactor 2, unusualness level 0.10, support minimum 0.5, and a fixed list of 40 calibration scores.
What this adds	Settings learned from labelled history with a certificate: with 95% confidence, at most 5% of the decisions they act on are wrong, if future cases are like the history. A monitor then watches that this still holds.
Demo outcome	On 4,000 fresh authorisations, hand-set settings act on 25.2% and are wrong on 6.9% of those; learned settings act on 29.8% and are wrong on 2.3%. A drift is caught two batches after it starts, once the monitor's window holds enough drifted outcomes.
Example id	`parameter_learning` (`uv run grounded-decision examples parameter_learning`)

Try it in your browser

The showcase site's the risk monitor page streams this example's six batches of live traffic through the monitor in your browser, with its bounds after every outcome; the parameter-learning page learns from this example's history. It runs this package itself, in Pyodide.

The shown numbers are the documented example, as the engine computed it; change a setting to compute it in your browser.

The risk monitor runs here when JavaScript is on; the risk monitor page has it too.

The decision today¶

The fraud policy is the one in Fraud review: the risk model sends suspicious or clean, the policy proposes block or pass, and four checks read the evidence, which is the flagged account's device neighbourhood: the accounts that logged in from the same phones and browsers, and each one's last review outcome. Each check has a setting:

Check	Setting	Hand-set value	Where it came from
Neighbour agreement	How far above 1 the evidence strength must be (the margin)	0.05, so strength above 1.05	A library default
Knife-edge	How wide the "too close to the cut-off" window is (the prefactor)	2.0	A library default
Unusual cases	The unusualness level α, and the calibration set it compares with	0.10, and 40 scores of recent confirmed-fraud cases	A default, and a list refreshed by hand
Support	The share of linked accounts that must carry the action's outcome	0.5	A library default (this demo's evidence does not measure support, so the check passes)

None of these says what error rate it buys. The calibration set has a second problem: it holds scores of confirmed fraud cases, but a clean authorisation proposing pass is compared with it too.

Where today's rule goes wrong¶

The demo builds 4,000 fresh authorisations that the learner never sees and decides each one three ways. "Wrong" means the proposed action turned out wrong: a blocked customer was genuine, or a passed card was fraud.

Settings	Act rate	Wrong among the acts
Hand-set, agreement measured the default way (`uniform`)	63.0%	18.6% (469 of 2,520)
Hand-set, agreement measured against the label mix (`label_mix`)	25.2%	6.9% (70 of 1,009)
Learned, `label_mix`	29.8%	2.3% (27 of 1,191)

Acting on every model call would be wrong 22.4% of the time. The hand-set settings remove most of that, but still miss the 5% the fraud lead asked for, and nobody could have known in advance. The learned settings act on more cases than the hand-set ones, and the error rate among the ones they act on is inside the target.

What the system learns, in your terms¶

You supply three things the data cannot:

The risk you accept: at most 5% of automatic decisions wrong, certified with 95% confidence (RiskTarget(max_risk=0.05, confidence=0.95)).
What a related case is: two accounts that used the same device.
What "correct" means: the proposed block was right if the account turned out to be fraud; the proposed pass was right if it did not.

From a labelled history the learner chooses everything else:

Setting	How it is learned
Calibration scores	The unusualness scores of past cases where the same proposed action turned out correct: blocks are compared with blocks that were right, passes with passes that were right.
Evidence-strength margin, knife-edge window, unusualness level, support minimum	Every combination in a grid of 1,200, including "this check off", is tried on part of the history. Only combinations whose error rate is shown to be below the target by an exact test are kept, and of those the one that acts most often is chosen.

Why agreement is measured against the label mix

Most accounts in a device neighbourhood are clean. Take 30 accounts, 3 confirmed fraud, and 75 links drawn at random, so the links say nothing about fraud. Because most links join two clean accounts, the default (uniform) agreement is 0.653 and the evidence strength 2.13: it would pass as strong evidence. Measured against the neighbourhood's own mix of labels (agreement_baseline="label_mix"), agreement is 0.037 and strength 0.01: correctly held. On the demo history, learning from uniform evidence certifies none of the 1,200 combinations: the inflated agreement no longer separates good cases from bad ones. This is the situation label_mix is for: an imbalance across the whole population and links that may be random. It is not the default, because it misjudges tight rings that are nearly all one label (see Build evidence).

Data you provide¶

Input	Source system	Refresh	Owner	Minimum / quality bar
Past flagged authorisations: the evidence each decision was made on, the proposed action	Your decision service's audit log (the reference service exports it at `GET /history.jsonl`)	When you re-learn	Fraud data engineering	One line per decision; keep one decision per cluster of related cases (see Limits).
Whether the proposed action was right	Chargebacks, review outcomes, customer contacts	As outcomes arrive	Fraud ops	Outcomes for held decisions too, not only the acted ones.
Whether each past decision was carried out (`was_acted_on`)	Your audit log	When you re-learn	Fraud data engineering	Lets the learner warn when only acted-on decisions have outcomes. The demo records what the hand-set module did.
The acceptable error rate and confidence	The fraud lead	When the business changes it	Fraud lead	A number you can defend; data cannot choose it.

In the demo the history is synthetic: synthetic_authorisations in src/grounded_decision/examples/parameter_learning.py generates raw (account_id, device_id) login rows for 3,000 neighbourhoods, and evidence_for turns them into evidence with snapshot_from_graph, as your evidence provider would.

Worked example¶

These numbers come from uv run grounded-decision examples parameter_learning.

What was learned (learned.explain()):

Goal: at most 5.0% wrong of the decisions it acts on, certified with 95% confidence.
History: 3000 labelled cases, split calibration 900, risk 1500, holdout 600 (seed 0, dataset data@83aa73d12869).
Tried 1200 combinations of settings; 18 were certified.
Procedure: fixed-sequence testing. The combinations were ordered on the calibration part (most clearly safe there first), then tested in that order on the risk part, each at significance 0.05, stopping at the first failure.
Chosen: reconstruction margin 0 (evidence strength must exceed 1), window prefactor 0 (knife-edge check off), decision line each case's own, conformal alpha 0.2, minimum support 0 (support check off).
Certificate: with 95% confidence the true error rate is at most 5.0%, provided future cases are independent draws from the same process as this history.
Risk part (the data the settings were chosen on, so optimistic): acted on 437 of 1500, 5 wrong.
Hold-out (never used for choosing): acted on 164 of 600 (27.3%), 5 wrong; error rate at most 6.30%.
Fingerprint: params@8560e292ea87

Read it as: once agreement is measured against the label mix, evidence strength above 1 (the Kesten–Stigum line itself, no extra margin) is enough; the knife-edge check did not help on this data, so it is off; the unusual-case check stays on, at 0.20, now with calibration scores per action; the support check is off because this demo's evidence does not measure support. The hold-out's bound (6.30% with 5 wrong of 164) is wider than the 5% target: with so few acts the hold-out is only a preview, and the certificate comes from the risk part's exact test. The fingerprint params@8560e292ea87 identifies these settings and their provenance; store it with every decision.

Three fresh authorisations decided with the learned settings:

Row	Event	Evidence strength	Unusualness p-value	Today's rule	Learned settings
1	`suspicious`	2.22 (above 1)	0.66 (above 0.20)	block	act `block`
2	`suspicious`	0.14: neighbours barely agree, below the learned 1	0.21	block	hold; proposed `block` kept
3	`clean`	1.05	0.065, unlike past correct passes (below 0.20)	pass	hold; proposed `pass` kept

Monitoring after launch. The demo then feeds six batches of 400 live authorisations to a RiskMonitor that keeps the last 300 acted-on outcomes. From batch 4 the world changes: the model's calls stop tracking the device graph (fraudsters start mimicking normal device sharing), so it is right 75% of the time whatever the evidence says.

Batch	Drifted	Wrong of acted-on	Error rate between (95%)	Monitor says
1	no	0 of 113	0.0% and 2.6%	`within_target`
2	no	1 of 215	0.0% and 2.2%	`within_target`
3	no	5 of 300	0.7% and 3.5%	`within_target`
4	yes	9 of 300	1.6% and 5.2%	`uncertain`
5	yes	14 of 300	2.8% and 7.2%	`uncertain`
6	yes	25 of 300	5.9% and 11.4%	`above_target`

In batches 4 and 5 the window still holds outcomes from before the drift, so the bounds climb but straddle 5% (uncertain). At batch 6 even the optimistic bound is above 5%: stop acting automatically and re-learn.

How the certificate is computed

The history is shuffled with a recorded seed and split three ways: 30% builds the calibration scores, 50% (the risk part) tests every combination of settings, and 20% (the hold-out) is only reported.

For each of the 1,200 combinations, the learner counts on the risk part how many cases it would act on (\(n\)) and how many of those were wrong (\(k\)), and computes the exact binomial p-value of "the true error rate is at least 5%": \(P(\text{Binomial}(n, 0.05) \le k)\). The combinations are tested in an order fixed in advance from the calibration part (most clearly safe there first); each is certified while its p-value is at most 0.05 (the 5% left by 95% confidence), and testing stops at the first one that is not (fixed-sequence testing). Of the certified combinations the one with the most acts is chosen. This is the Learn-then-Test procedure (Angelopoulos, Bates, Candès, Jordan and Lei, 2021).

The monitor's bounds are exact (Clopper–Pearson) one-sided 95% bounds on the error rate among the last 300 acted-on outcomes. above_target means the lower bound is above the target.

What happens next¶

flowchart LR
    D[Decide] --> O[Record outcomes, acted and held]
    O --> E[Export history]
    E --> L[Learn with your risk target]
    L --> C{Certified?}
    C -- no --> M[More history or a higher target; keep current settings]
    C -- yes --> R[Restart with the learned parameters]
    R --> W[Monitor the live error rate]
    W -- above target --> S[Stop acting automatically: shadow mode]
    S --> E
    W -- within target --> D

Signal	Who is told	What they see	What they do
Learning certified	Fraud data science	`explain()` lines and the `params@…` fingerprint	Review, then deploy the parameters file.
Nothing certified	Fraud data science	"No combination could be certified" and how many acted-on cases are needed	Keep the current settings; collect more outcomes or agree a higher risk with the fraud lead.
Monitor `above_target`	On-call and the fraud lead	Error-rate bounds above the target	Stop automatic actions (the reference service switches to shadow by itself), find what changed, re-learn.

Tuning and expected workload¶

Setting	Policy or data?	How to choose	Too tight	Too loose
Acceptable error rate (`max_risk`, 5%)	Policy	The cost of a wrong automatic block or pass against the cost of a review	Nothing certifies, or very few acts	Errors you would not accept from a person
Confidence (95%)	Policy	How sure the certificate must be	Needs more history	The certificate is weak
History size (3,000)	Data	Enough acted-on cases in the risk part: even with no errors, 5% at 95% needs at least 59 for the combination being tested (197 with `--procedure bonferroni` over 1,200 combinations)	—	Nothing certifies
Monitor window (300 acted-on outcomes)	Policy	Long enough to be sure, short enough to notice drift quickly	Noisy alarms	Slow to notice

Expected workload. Here the learned settings act on more cases than the hand-set ones (29.8% against 25.2%), so fewer go to reviewers, and they are wrong less often. That is not guaranteed: on other histories meeting the target costs acts. If the price is too high, agree a different max_risk and re-learn rather than editing a setting by hand.

Limits, risks and compliance¶

Same process assumed. The certificate holds if future cases are drawn from the same process as the history. A new fraud pattern, a new model version or a new market breaks it; the monitor is how you find out.
Independent cases assumed. Decisions about accounts in the same fraud ring share their evidence and usually their outcome, so fifty of them confirm one fact fifty times. Keep one decision per cluster of related cases in the history. The demo's neighbourhoods are separate by construction.
Selective labels. If only acted-on decisions get outcomes, the history covers only the cases the old settings liked. Record outcomes for held decisions too, and record was_acted_on: the learner then warns when every case with a known outcome was acted on.
Re-learn, don't patch. The calibration scores and the check settings were certified together. Swapping in fresh calibration scores without re-learning changes the configuration the certificate is about.
Direction is not checked in this demo: its evidence does not measure support, so the learner switches the support check off. With support measured, as in Fraud review, the learner chooses the support minimum like the other settings.
Synthetic data. The neighbourhoods, the model and the drift are generated with fixed seeds to make the effect visible. Your own numbers will differ; only your history can tell you what your settings should be.
The monitor checks a fixed bound repeatedly, which raises the chance of a false alarm over time. Treat one above_target as a reason to investigate and re-learn.

Wiring it in¶

Record outcomes. For every decision, store the evidence it was made on, the proposed action and, later, whether that action was right. The reference service does this with POST /outcomes and exports GET /history.jsonl.
Learn from the command line, with your target:

make learn HISTORY=history.jsonl MAX_RISK=0.05

or from Python:

from grounded_decision.learn import HistoricalCase, RiskTarget, learn_parameters

learned = learn_parameters(history_cases, RiskTarget(max_risk=0.05, confidence=0.95))
print("\n".join(learned.explain()))

Build the module once with the learned settings, and complete each request's evidence with the calibration scores for its proposed action:

module = learned_module(learned)  # Module(..., limits=learned.limits())
proposed = module.peek_proposed_action(event.name)
record = module.decide(event, learned.apply(evidence, proposed))

Monitor: RiskMonitor(learned.target).record(acted=..., action_was_correct=...) for every outcome of a decision made with these settings; stop acting automatically on above_target.

Learn the settings from your outcomes covers the procedure in detail.

Run it yourself¶

uv run grounded-decision examples parameter_learning
uv run grounded-decision examples parameter_learning --json

The text report prints the learning results under setup, then one block per showcased decision. How to read those blocks: Example library: reading a report.