Skip to content

Learn parameters from your history

The checks have settings: how strong the related cases must be, how wide the knife-edge window is, how unusual a case may look, how many related cases must point the same way as the action, and which recent cases count as "typical". Guessing them, or copying the defaults, gives you a service whose error rate nobody can state. This page shows how to learn them instead from past decisions whose outcomes you know, so that each setting comes with a certified error rate and a record of where it came from.

Who this is for: the data scientist or engineer configuring a service before launch, and whoever re-learns the settings after it. The domain owner needs only steps 1 and 5.

What you need:

  • A policy and an evidence builder (see From your data to evidence).
  • Labelled history: past decisions, each with the evidence you had at the time, the action the policy proposed, and whether that action later turned out right. Step 2 says how many.
  • uv sync done in this repository, or grounded-decision installed.

What you decide, and what is learned. You choose the error rate you accept (the risk target) and what counts as a related case and as a correct outcome. Everything else on this page is measured from your graph or learned from your history. Where every number comes from lists every value and its source.

Try it in your browser

The showcase site's the parameter-learning page learns these settings in your browser on this page's 3,000-case history (or a CSV of your own, read in the browser and never uploaded), with every combination's exact test plotted; its risk monitor page runs step 8's stream. It runs this package itself, in Pyodide.

The shown numbers are the documented example, as the engine computed it; change a setting to compute it in your browser.

The parameter learner runs here when JavaScript is on; the parameter-learning page has it too.

The risk monitor runs here when JavaScript is on; the risk monitor page has it too.

How it works, in one paragraph

The learner splits your history into three parts. From the first (calibration split) it builds the calibration scores: the unusualness scores of past cases where each proposed action turned out right. On the second (risk split) it tries every combination of settings in a grid (1,200 by default, including "this check off"), counts how often each combination would have acted and how often those acts were wrong, and keeps only the combinations whose error rate an exact statistical test certifies is below your target. By default it tests them one after another in an order fixed in advance from the calibration split (most clearly safe first) and stops at the first failure; this fixed sequence needs no correction for the number of combinations. Of the certified combinations, it picks the one that acts most often. The third part (hold-out) is never used for choosing; it reports how the chosen settings do on cases the learner did not see. The method is called Learn-then-Test; the theory page states its guarantee precisely.

Step 1: choose the risk target

The risk target is the one number data cannot choose for you, because it is your risk appetite. It has three parts:

Part Meaning Example
max_risk The largest error rate you accept. 0.05: at most 5% wrong
measure Which error rate: error_rate_among_acts (of the decisions the service acts on, the share that are wrong; the default) or false_act_rate (of all decisions, the share that act and are wrong). "At most 5% of automatic blocks are wrong"
confidence How sure the certificate must be. 0.95: wrong at most 1 time in 20

Write the target in the domain owner's words in your design review: "at most 5% of automatic blocks are on customers who were not committing fraud, certified at 95% confidence".

Step 2: check you have enough history

Even a perfect record certifies little if it is short: with 50 acts and no errors, the 95% upper bound on the true error rate is still 5.8%. minimum_trials_to_certify(max_risk, 1 - confidence, tests) gives the fewest acted-on cases in the risk split that could certify a target, with zero errors. With the default fixed-sequence procedure each combination is tested at the full significance, so tests is 1; with --procedure bonferroni the significance is divided by the grid size (1,200):

from grounded_decision.learn import minimum_trials_to_certify

for max_risk in (0.10, 0.05, 0.02, 0.01):
    print(
        max_risk,
        minimum_trials_to_certify(max_risk, 1 - 0.95, 1),
        minimum_trials_to_certify(max_risk, 1 - 0.95, 1200),
    )

0.1 29 96
0.05 59 197
0.02 149 500
0.01 299 1004

Those are floors. If some acts are wrong you need more: at a 5% target (95% confidence, fixed sequence), 59 acts if a quarter of the target rate is wrong (1.25%), 153 if half is (2.5%). For a 2% target the same figures are 149 and 386. (With Bonferroni over 1,200 combinations: 371 and 953 at 5%, 937 and 2,470 at 2%.) The fixed-sequence floor applies to the combination being tested; if the first combinations in the order act on too few cases, testing stops early, so leave a margin.

To turn acted-on cases into total history, divide by the share of cases the service will act on and by the share in the risk split (half, by default). If you expect to act on about 20% of cases and want a 5% target, you need at least \(153 / (0.2 \times 0.5) \approx 1{,}500\) labelled cases to leave room for some errors. The worked example below uses 3,000.

Step 3: export your history as JSON Lines

Write one line per past decision in which the policy proposed an action:

{"evidence": {"n": 12, "p_hat": 0.569, "p_threshold": 0.2, "degree": 5.36, "lambda_hat": 0.903, "n_states": 2, "scores": [], "label_score": 0.312, "observations": {}, "support": null}, "proposed_action": "pass", "action_was_correct": true, "case_id": "case-0", "was_acted_on": false}
Field What to put there
evidence EvidenceSnapshot.to_dict() of the evidence as it was when the decision was made: the graph rebuilt as of that day, never including outcomes learned afterwards. scores may be empty; the learner builds its own.
evidence.label_score This case's unusualness score, computed exactly as production will compute it (for example 1 − the model's probability that the proposed action is right, from the model version you will run).
proposed_action What the policy proposed ("block", "pass").
action_was_correct Whether that action turned out right: the review outcome, the chargeback, the 72-hour follow-up. Include cases that were held or reviewed, not only cases that were acted on (see pitfalls).
evidence.support Optional: the share of related cases carrying the outcome the proposed action implies, as it was at decision time (null if not measured). Lets the learner choose a support minimum.
case_id Optional, kept for audit.
was_acted_on Optional but recommended: whether the decision was carried out (true) or held (false). The learner warns when every case with a known outcome was acted on (selective labels) and, more softly, when no case records it.

If you already run the reference service in shadow mode, its audit table has the evidence (evidence_json) and the proposed action for every decision; join the outcomes you learned later (Roll out safely).

To follow this page without your own data, this script writes a synthetic history of 3,000 fraud-review decisions. Strong, clear, ordinary-looking evidence is usually right; thin, knife-edge or unusual evidence is often wrong. It draws through grounded_decision.examples.portable_random rather than numpy's uniform and math.exp, which can differ in the last bit between processors, so every machine writes the same file and the same dataset fingerprint. Save it as make_history.py:

"""Write a synthetic labelled history to history.jsonl (stand-in for your export)."""

import json
import sys

import numpy as np

from grounded_decision.examples.portable_random import logistic, uniform
from grounded_decision.learn import HistoricalCase
from grounded_decision.types import EvidenceSnapshot

case_count = int(sys.argv[1]) if len(sys.argv) > 1 else 3000
generator = np.random.default_rng(1)
with open("history.jsonl", "w", encoding="utf-8") as history:
    for index in range(case_count):
        degree = uniform(generator, 0.5, 10.0)
        agreement = uniform(generator, 0.0, 0.95)
        entity_count = int(generator.integers(8, 150))
        density = uniform(generator, 0.0, 0.6)
        unusualness = uniform(generator, 0.0, 1.0)
        # Strong, clear, ordinary-looking evidence is usually right.
        log_odds = (
            1.6 * (degree * agreement**2 - 1.0)
            + 6.0 * abs(density - 0.2)
            - 3.0 * unusualness
            + 1.0
        )
        probability_right = logistic(log_odds)
        case = HistoricalCase(
            evidence=EvidenceSnapshot(
                n=entity_count,
                p_hat=density,
                p_threshold=0.2,
                degree=degree,
                lambda_hat=agreement,
                label_score=unusualness,
            ),
            proposed_action="block" if index % 2 else "pass",
            action_was_correct=bool(generator.random() < probability_right),
            case_id=f"case-{index}",
        )
        history.write(json.dumps(case.to_dict()) + "\n")
print(f"wrote {case_count} cases to history.jsonl")

uv run python make_history.py

wrote 3000 cases to history.jsonl

Step 4: learn

uv run grounded-decision learn history.jsonl --max-risk 0.05 --output params.json

Goal: at most 5.0% wrong of the decisions it acts on, certified with 95% confidence.
History: 3000 labelled cases, split calibration 900, risk 1500, holdout 600 (seed 0, dataset data@6575a29dabe3).
Tried 1200 combinations of settings; 210 were certified.
Procedure: fixed-sequence testing. The combinations were ordered on the calibration part (most clearly safe there first), then tested in that order on the risk part, each at significance 0.05, stopping at the first failure.
Chosen: reconstruction margin 2 (evidence strength must exceed 3), window prefactor 0 (knife-edge check off), decision line each case's own, conformal alpha 0 (unusual-case check off), minimum support 0 (support check off).
Certificate: with 95% confidence the true error rate is at most 5.0%, provided future cases are independent draws from the same process as this history.
Risk part (the data the settings were chosen on, so optimistic): acted on 280 of 1500, 5 wrong.
Hold-out (never used for choosing): acted on 120 of 600 (20.0%), 1 wrong; error rate at most 3.89%.
Warning: unknown whether held decisions are represented (no case records was_acted_on); if outcomes exist only for acted-on decisions, the certificate describes that filtered population
Fingerprint: params@e55f023199be
Wrote params.json

The command exits with 0 when a setting was certified and 2 when none was, so a scheduled job can refuse to deploy an uncertified result. Options: --confidence 0.99, --measure false_act_rate, --seed 3, --procedure bonferroni (test every combination at the significance divided by the grid size, instead of the default fixed sequence), and --learn-decision-line to also choose p_threshold from quantiles of the observed link density (this multiplies the grid by 6). The same run as a make target, which reads HISTORY (default history.jsonl) and writes PARAMETERS (default parameters.json):

make learn MAX_RISK=0.05

From Python, the same call is learn_parameters(cases, RiskTarget(max_risk=0.05), seed=0); pass grid=ParameterGrid(...) to try other candidate values and procedure="bonferroni" for the other procedure.

On the same history, --procedure bonferroni certifies 12 combinations instead of 210 and chooses one that acts on 15.7% of the hold-out instead of 20.0% (1 wrong in 94, at most 4.95%). Both procedures give the same guarantee; the theory page explains why the fixed sequence certifies more.

Step 5: read the result

Read the printout line by line. Each line answers a question an auditor will ask.

Line What it tells you
Goal The risk target, restated. Check it is the one the domain owner signed.
History How many cases, how they were split, the random seed, and a fingerprint of the exact history file (data@6575a29dabe3). Anyone with the same file and seed gets the same result.
Tried 1,200 combinations; 210 had an error rate certified below 5%. Few certified combinations means the target is near what your evidence can support.
Procedure How the combinations were tested. Fixed sequence (the default): in an order computed from the calibration split only, each at the full significance 0.05, stopping at the first failure.
Chosen The settings. Margin 2: evidence strength \(d\lambda^2\) (average links per case × neighbour agreement²) must exceed 3, much stricter than the default 1.05. Window prefactor 0, alpha 0 and minimum support 0 switch the knife-edge, unusual-case and support checks off: on this history they held cases without preventing errors (this synthetic history does not measure support at all).
Certificate The guarantee itself: with 95% confidence, at most 5% of acted-on decisions are wrong, if future cases are independent draws from the same process.
Risk part On the cases used for choosing, the chosen settings acted on 280 and 5 were wrong. These are the cases the choice was made on, so the counts are optimistic; the certificate above is the guarantee.
Hold-out On 600 cases never used for choosing, it acted on 20.0% with 1 wrong; the error rate is at most 3.89% with 95% confidence. This is the honest preview of production.
Warning The history does not say which decisions were carried out and which were held (was_acted_on), so the learner cannot rule out selective labels. Add the field to your export (step 3).
Fingerprint params@e55f023199be: a short hash of the settings and everything below. Store it on every decision record (step 7).

params.json holds the same information in full. Its parameters section is what the module uses: the four check settings (reconstruction_margin, operational_fk_prefactor, conformal_alpha, minimum_support), the decision line (null means each case keeps its own p_threshold), and the calibration scores per proposed action (280 for block, 302 for pass here). Its provenance section is the audit trail:

"provenance": {
  "configurations_certified": 210,
  "configurations_tried": 1200,
  "dataset_fingerprint": "data@6575a29dabe3",
  "grid": {"conformal_alphas": [0.0, 0.02, 0.05, 0.1, 0.2], "decision_lines": [null],
           "minimum_supports": [0.0, 0.5, 0.6, 0.7, 0.8, 0.9],
           "reconstruction_margins": [-1.0, 0.0, 0.05, 0.25, 0.5, 1.0, 2.0, 4.0],
           "window_prefactors": [0.0, 0.5, 1.0, 2.0, 4.0]},
  "holdout": {"act_rate": 0.2, "acts": 120, "cases": 600,
              "error_rate_upper_bound": 0.0389208468602, "errors": 1},
  "method": "learn-then-test (exact binomial p-values, fixed-sequence testing ordered on the calibration part)",
  "minimum_risk_cases_needed": 59,
  "procedure": "fixed_sequence",
  "risk_split": {"act_rate": 0.18666666666666668, "acts": 280, "cases": 1500,
                 "error_rate_upper_bound": 0.0371778866575, "errors": 5},
  "seed": 0,
  "split_sizes": {"calibration": 900, "holdout": 600, "risk": 1500},
  "target": {"confidence": 0.95, "max_risk": 0.05, "measure": "error_rate_among_acts"},
  "warnings": ["unknown whether held decisions are represented (no case records was_acted_on); if outcomes exist only for acted-on decisions, the certificate describes that filtered population"]
}

(Reformatted onto fewer lines; the file itself is indented JSON.) The grid shows which values were possible: margin -1.0, prefactor 0.0, alpha 0.0 and minimum support 0.0 switch a check off, so the learner can find that a check does not help on your data. Here it kept only neighbour agreement on, at a strict margin.

Show the domain owner the hold-out line: "on 600 past cases it had not seen, it would have acted on 20% and been wrong once". The rest of the decisions (80%) go to people; size the review queue with Estimate impact.

Step 6: load the settings into your module

learned.limits() returns Limits carrying the four learned check settings (reconstruction_margin, operational_fk_prefactor, conformal_alpha, min_support); learned.apply(evidence, proposed_action) fills in the calibration scores for that action (and the decision line, if one was learned). apply needs the proposed action before you decide, so ask the policy's chart what it would propose. Save as use_params.py:

import dataclasses
import json
from pathlib import Path

from grounded_decision.examples.common import two_path_module
from grounded_decision.learn import LearnedParameters
from grounded_decision.types import Event, EvidenceSnapshot

learned = LearnedParameters.from_dict(json.loads(Path("params.json").read_text()))
if not learned.certified:
    raise SystemExit("parameters not certified; every decision would be held")

module = two_path_module("fraud_review", "clean", "suspicious", "pass", "block")
module = dataclasses.replace(module, limits=learned.limits())


event = Event("suspicious")
evidence = EvidenceSnapshot(
    n=40,
    p_hat=0.45,
    p_threshold=0.2,
    degree=6.0,
    lambda_hat=0.85,
    label_score=0.30,
)
action = module.peek_proposed_action(event.name)  # no state change
evidence = learned.apply(evidence, action)
record = module.decide(event, evidence)
row = {**record.to_dict(), "parameters_version": learned.fingerprint}
print(
    action,
    len(evidence.scores),
    row["verdict"],
    row["action"],
    row["parameters_version"],
)
print(record.reason)

block 280 act block params@e55f023199be
all channels passed

Evidence strength here is \(6 \times 0.85^2 = 4.3\), above the learned bar of 3. With 3 links per case instead of 6 (strength 2.17) the same call abstains with the reason d λ² = 2.1675 vs 1+2.000 (at or below KS; abstain); with the default settings (Limits()) it acts.

Pass learned.limits(base) if your module already has non-default size caps; only the four learned fields are replaced. require_support is kept from base: leave it off unless every decision's evidence measures support. Use the calibration scores from apply as they are. If you refresh them on your own schedule, the certificate no longer describes what runs; re-learn instead (step 8).

Step 7: record where the settings came from

Keep params.json under version control next to your policy, and store learned.fingerprint on every decision record, as parameters_version above. Months later, an auditor can go from any decision to the exact settings, the history they were learned from (dataset_fingerprint), the target they were certified for, and the hold-out result.

New learned settings change what the service accepts, so ship them like any setting change: shadow first, then enforce. In the reference service they change the settings version (parameters_fingerprint), not the policy version, so saved states stay valid (Roll out safely). The reference service's own learn step is shown in Build a decision service.

Step 8: monitor, and re-learn

The certificate holds while new cases come from the same process as the history. When the world changes (a new fraud pattern, a retrained model, a new hospital), it stops applying, and nothing in the certificate tells you so. RiskMonitor watches the realised error rate of recent decisions whose outcome is now known:

from grounded_decision.learn import RiskMonitor, RiskTarget

monitor = RiskMonitor(RiskTarget(max_risk=0.05), window=500)
# 300 acted-on decisions with 1% wrong, like the certified history ...
for index in range(300):
    monitor.record(acted=True, action_was_correct=index % 100 != 0)
print(
    monitor.status().value,
    monitor.errors,
    monitor.trials,
    f"{monitor.upper_bound():.3f}",
)
# ... then the world changes: the next 300 are 20% wrong.
for index in range(300):
    monitor.record(acted=True, action_was_correct=index % 5 != 0)
print(
    monitor.status().value,
    monitor.errors,
    monitor.trials,
    f"{monitor.lower_bound():.3f}",
)

within_target 3 300 0.026
above_target 62 500 0.100
Status Meaning What to do
insufficient_data No outcomes in the window yet. Wait.
within_target Even the pessimistic bound on the recent error rate is at or below the target. Nothing.
uncertain Neither confidently below nor confidently above. Keep watching; normal soon after launch.
above_target Even the optimistic bound is above the target (10.0% > 5% here). Stop acting automatically (hold everything), investigate, re-learn.

Use the same RiskTarget you learned with. With the default measure, record ignores decisions that were not acted on. Because the monitor is checked again and again, a single above_target can occasionally be a false alarm: treat it as a reason to investigate and re-learn, not as proof.

Re-learn on a schedule as well (monthly, or at the outcome cycle of your domain), on the most recent history, and always after a model retrain or a change in what counts as a related case. Compare the new hold-out line with the old one before deploying.

When nothing is certified

The same 3,000 cases certify a 2% target (24 combinations; 13.8% of the hold-out acted on). With a 0.5% target:

uv run grounded-decision learn history.jsonl --max-risk 0.005

Goal: at most 0.5% wrong of the decisions it acts on, certified with 95% confidence.
History: 3000 labelled cases, split calibration 900, risk 1500, holdout 600 (seed 0, dataset data@6575a29dabe3).
Tried 1200 combinations of settings; 0 were certified.
Procedure: fixed-sequence testing. The combinations were ordered on the calibration part (most clearly safe there first), then tested in that order on the risk part, each at significance 0.05, stopping at the first failure.
No combination could be certified, so these settings hold every decision. Either there is too little history (even with zero errors, at least 598 acted-on cases are needed in the risk part), or every setting that acts often enough makes too many mistakes. Fixes: more labelled history, a better definition of related cases or a better model score, or a higher acceptable risk.
Warning: unknown whether held decisions are represented (no case records was_acted_on); if outcomes exist only for acted-on decisions, the certificate describes that filtered population
Warning: the first combination tested (the most clearly safe one on the calibration part) failed on the risk part, so testing stopped: the target is too strict for this history, or there is too little history to show it is met
Fingerprint: params@ecb597c52b55

The exit code is 2, and the returned settings fail closed: the margin is set so high (\(10^{12}\)) that every decision is held. The last warning is specific to the fixed sequence: the combination that looked safest on the calibration split failed, so nothing after it was tested. With only 300 cases (uv run python make_history.py 300) at 5%, nothing is certified either, and a further warning explains part of why:

Warning: fewer than 49 calibration scores for block, pass; the unusual-case check cannot hold them at alpha 0.02

Your options, in the order to try them:

  1. Collect more labelled history. The printout says how many acted-on cases the risk split needs at minimum.
  2. Accept a higher risk, if the domain owner agrees: the 3,000 cases above certify 2% but not 0.5%.
  3. Use a smaller grid, or a different procedure. With --procedure bonferroni, fewer candidates divide the confidence fewer ways: 54 combinations instead of 1,200 lower the zero-error floor at 2% from 500 to 346 acts (minimum_trials_to_certify(0.02, 0.05, 54)). With the fixed sequence, a grid without settings you would never deploy keeps unpromising combinations from stopping the sequence early. Choose the grid and the procedure before looking at the result, not after.
  4. Improve the evidence. If even large histories certify nothing, the related cases do not separate right from wrong decisions: revisit what counts as a related case and a label (Build evidence).

Do not deploy the uncertified file as if it were a result. It holds everything, which is safe but sends every decision to people.

Result

You have settings learned from your own outcomes, a certificate that their error rate is at most your target if the future resembles the history, a hold-out estimate of the act rate, a provenance record and a fingerprint on every decision, and a monitor that tells you when to re-learn.

What can go wrong

  • Selective labels. If you only learn outcomes for some decisions (only acted cards get chargebacks; only held cases get a reviewer), the history describes a different population from the one you will decide on, and the certificate is about that population. Record outcomes for held decisions too, sample acted ones for review, and export was_acted_on: when every case with a known outcome was acted on, the learner warns every case with a known outcome was acted on; outcomes of held decisions are missing, so the certificate describes a filtered population.
  • Drift. The guarantee assumes future cases come from the same process as the history. Watch RiskMonitor and re-learn on a schedule.
  • label_score computed differently. If history used one model version or formula and production uses another, the calibration scores and the unusual-case check describe the wrong thing. Recompute the history's scores with the production model.
  • A random split hides time. The learner shuffles the history before splitting, so its hold-out comes from the same months as the training data. To see whether the settings survive time, learn on older cases and test on the most recent yourself:
import json
from pathlib import Path

import numpy as np

from grounded_decision.learn import (
    HistoricalCase,
    RiskTarget,
    case_arrays,
    learn_parameters,
    predicted_acts,
    upper_confidence_bound,
)

cases = [
    HistoricalCase.from_dict(json.loads(line))
    for line in Path("history.jsonl").read_text().splitlines()
]  # oldest first
older, recent = cases[:2400], cases[2400:]
target = RiskTarget(max_risk=0.05)
learned = learn_parameters(older, target)

arrays = case_arrays(recent, learned.calibration_scores)
acts = predicted_acts(
    arrays,
    margin=learned.reconstruction_margin,
    prefactor=learned.operational_fk_prefactor,
    decision_line=learned.decision_line,
    alpha=learned.conformal_alpha,
    minimum_support=learned.minimum_support,
)
errors = int(np.count_nonzero(acts & arrays.wrong))
acted = int(np.count_nonzero(acts))
bound = upper_confidence_bound(errors, acted, target.confidence)
print(
    f"{learned.status}; recent {len(recent)} cases: acted on {acted}, {errors} wrong,"
    f" error rate at most {bound:.2%}"
)

certified; recent 600 cases: acted on 96, 1 wrong, error rate at most 4.85%

(The synthetic history has no time trend; on yours, a recent bound above the target is a sign of drift.)

  • Seed shopping. Different seeds split the history differently and can choose different settings, all equally valid: seed 0 above chose alpha 0 (unusual-case check off), --seed 1 chooses alpha 0.02 with the same margin, prefactor and support minimum. Pick the seed before you run and record it. Re-running with new seeds until you like the answer quietly spends the confidence: each extra try is another chance for a bad setting to slip through.
  • Labels that leak the answer. Evidence in the history must be what was known at decision time. A graph rebuilt with today's outcomes makes past decisions look easier than they were, and the learned settings too permissive.
  • Blind spots stay in the numbers. Without a support share in the evidence, no check looks at the direction of the evidence (It acts on the wrong side). The certificate covers the whole pipeline as it ran in your history, so such errors are counted in the measured error rate, not hidden; but if your event source changes, so does that rate.

Next