Learning settings with a guarantee (Learn-then-Test)¶

Every data check has a setting, and a setting chosen by hand has no error rate attached to it. This page states the result the library uses to choose the settings from labelled history instead, so that the chosen settings come with a certificate: with high probability, their error rate is at most a target you set. It covers the exact test, why picking the most useful certified setting keeps the guarantee, the two ways of testing many settings at once (fixed sequence, the default, and Bonferroni), what the guarantee assumes and does not cover, and how conservative it is.

It assumes you know what a p-value and a binomial distribution are. The step-by-step procedure is in Learn parameters from your history; the plain-language map of which numbers are learned is Where every number comes from. This is standard statistics; no result from the project's math catalog is involved.

Try it in your browser

The showcase site's the parameter-learning page plots the exact p-value of every combination in testing order against the significance line, for either procedure, so you can watch the fixed sequence stop at its first failure. It runs this package itself, in Pyodide.

The shown numbers are the documented example, as the engine computed it; change a setting to compute it in your browser.

The parameter learner runs here when JavaScript is on; the parameter-learning page has it too.

The situation¶

A fraud team has 3,000 past decisions. For each, it knows the evidence at the time, the action the policy proposed (block or pass) and whether that action later turned out right. It wants settings for the four data checks (the reconstruction margin, the knife-edge window prefactor, the conformal alpha, the support minimum) such that at most 5% of the decisions the service acts on are wrong, and it wants to be 95% sure of that.

The naive approach is to try many settings on the history and keep the one that looks best. The trouble is the winner's curse: among 1,200 candidates, the one with the best-looking record is partly lucky, so its record overstates how good it is.

Intuition: test every candidate, then choose among the survivors¶

Turn the question around. For each candidate setting, ask a yes/no question that can be answered with a known chance of being wrong: "is this setting's error rate below 5%?" Answer them so that the chance of any wrong "yes" across all candidates is at most 5%. Then every setting that got a "yes" is, with 95% probability, genuinely below 5%, all at once. Choosing among them, by any rule at all, cannot pick a bad one unless that 5% event already happened.

There are two ways to keep the chance of any wrong "yes" at 5%. One is to make each answer much stricter (Bonferroni: divide the 5% by the number of candidates). The other is to ask the questions in a fixed order decided beforehand, each at the full 5%, and stop at the first "no" (fixed sequence). The second works because a wrong "yes" can only happen after every earlier question was answered "yes"; it pays off when the order puts the safe candidates first. The library orders the candidates using a separate part of the history, so the order does not peek at the data the questions are answered on.

This is the idea of Learn-then-Test: replace "estimate the error, then pick" with "test each candidate's error against the target, then pick among the ones that pass".

Worked example¶

From the how-to's run (uv run grounded-decision learn history.jsonl --max-risk 0.05, Learn parameters from your history), the chosen setting (margin 2, the other three checks off) acted on 280 of the 1,500 cases in the risk split, and 5 of those acts were wrong.

If that setting's true error rate were exactly 5%, the chance of seeing 5 or fewer errors in 280 acts would be

\[ \sum_{i=0}^{5} \binom{280}{i}\, 0.05^{i}\, 0.95^{280-i} = 0.0047 . \]

That is the p-value. Under the default fixed-sequence procedure each test passes at the full \(1 - 0.95 = 0.05\), so this setting is certified, and so were all 209 settings tested before it in the order; 210 were certified in all, and this one acts most often. Under Bonferroni, each test would have to pass at \(0.05 / 1200 \approx 0.0000417\), which this setting misses; Bonferroni certifies only 12 settings on the same history and chooses one that acts on 210 risk cases with 0 wrong (\(p = 0.95^{210} \approx 0.000021\)).

On the 600 hold-out cases, which played no part in the choice, the fixed-sequence choice acted on 120 (20.0%) with 1 wrong; the Bonferroni choice acted on 94 (15.7%) with 1 wrong.

The precise statement¶

Data. Labelled cases \(Z_1, \dots, Z_N\), drawn independently from one distribution \(P\). The learner shuffles them with a recorded seed and splits them into a calibration part, a risk part of size \(m\), and a hold-out part.

Candidates. A finite grid \(\Lambda = \{\lambda_1, \dots, \lambda_K\}\) of settings (by default \(K = 8 \times 5 \times 1 \times 5 \times 6 = 1200\): margins, window prefactors, decision lines, conformal alphas, support minimums). The calibration part fixes the calibration scores; after that, each \(\lambda_j\) is a fixed rule that says, for any case \(z\), whether the module acts, \(A_j(z) \in \{0, 1\}\). Let \(W(z) \in \{0, 1\}\) be 1 when the proposed action was wrong.

Risk. For the default measure (error rate among acts) and the alternative (false-act rate):

\[ R(\lambda_j) = P\bigl(W = 1 \mid A_j = 1\bigr) \qquad\text{or}\qquad R(\lambda_j) = P\bigl(A_j = 1,\ W = 1\bigr). \]

Hypotheses. For a target \(\alpha\) (max_risk) and confidence \(1 - \delta\), test for each \(j\) the null hypothesis \(H_j : R(\lambda_j) \ge \alpha\) ("this setting is not good enough").

Exact binomial p-value. On the risk part, let \(k_j\) be the number of trials (acts, for the default measure; all \(m\) cases for the false-act rate) and \(e_j\) the number of acted-and-wrong cases. Then

\[ p_j = P\bigl(\mathrm{Bin}(k_j, \alpha) \le e_j\bigr) = \sum_{i=0}^{e_j} \binom{k_j}{i} \alpha^{i} (1-\alpha)^{k_j - i}, \qquad p_j = 1 \text{ if } k_j = 0 . \]

Given the calibration part and \(k_j\), the acted cases are independent draws from \(P\) conditioned on acting, so \(e_j \sim \mathrm{Bin}(k_j, R(\lambda_j))\). Under \(H_j\) this is stochastically at least \(\mathrm{Bin}(k_j, \alpha)\), hence \(P(p_j \le u) \le u\) for every \(u\): \(p_j\) is a valid p-value. No large-sample approximation is used (learn.statistics.binomial_cdf, checked against brute force in tests/test_learn.py). The tail is computed exactly: the target \(\alpha\) is a float, so exactly \(a / 2^m\), and \(p_j\) is a ratio of integers. The learner certifies by comparing that ratio with \(\delta\) (or \(\delta / K\)) as rational numbers, and reports \(p_j\) rounded up to the next float, so no rounding can certify a setting the exact test would reject, and every platform makes the same choice.

Family-wise control, fixed sequence (the default, procedure="fixed_sequence"). Before looking at the risk part, order the candidates \(\lambda_{(1)}, \lambda_{(2)}, \dots, \lambda_{(K)}\) using the calibration part only. Then certify \(\lambda_{(1)}, \lambda_{(2)}, \dots\) in that order while \(p_{(j)} \le \delta\), and stop at the first \(j\) with \(p_{(j)} > \delta\). Conditional on the calibration part, the order is fixed and the risk-part p-values are valid. Let \(j_0\) be the position of the first candidate whose null \(H_{(j_0)}\) is true. Certifying any bad candidate requires certifying every candidate before it, in particular \(\lambda_{(j_0)}\), which happens with probability \(P(p_{(j_0)} \le \delta) \le \delta\).

The order the library uses: for each candidate, the p-value it would get on the risk part if the risk part looked like the calibration part (its calibration acts and errors scaled to the risk part's size), from the smallest (most clearly safe) to the largest; ties go to the strictest candidate (fewest calibration acts), then grid order (_testing_sequence in learn/calibrate.py). Ordering by act rate alone (strictest first) is also valid, but strict candidates act on so few cases that a safe one can fail for lack of trials and stop the sequence early; on the 3,000-case synthetic history of the tests it acted on 6.9% of fresh cases, against 15.7% for Bonferroni. With the predicted p-value order, the same history gives 18.2%.

Family-wise control, Bonferroni (procedure="bonferroni"). Certify \(\lambda_j\) when \(p_j \le \delta / K\). By the union bound, the probability that any setting with \(R(\lambda_j) \ge \alpha\) is certified is at most \(\sum_j \delta / K = \delta\).

Both arguments are in the learn_parameters docstring. The procedure used is recorded in the provenance (procedure) and printed by explain() (the Procedure: line).

Result (Angelopoulos, Bates, Candès, Jordan and Lei, 2021, "Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control"). Let \(\hat\Lambda\) be the set of certified settings. Then

\[ P\Bigl(R(\lambda) < \alpha \ \text{for every}\ \lambda \in \hat\Lambda\Bigr) \ge 1 - \delta . \]

Consequently, for any setting \(\hat\lambda \in \hat\Lambda\) chosen by any rule, even one that looks at the same risk data, \(P\bigl(R(\hat\lambda) < \alpha\bigr) \ge 1 - \delta\). The probability is over the draw of the history (and the split).

Why choosing the most-acting certified setting is allowed. The guarantee is about the whole set \(\hat\Lambda\): outside an event of probability at most \(\delta\), every member is below the target. The library picks the member with the most acts (ties broken by fewer errors, then grid order). Whatever it picks, it picks from a set that is entirely good on that event, so the selection costs nothing. This is what family-wise control buys over testing a single pre-chosen setting.

When nothing is certified. If \(\hat\Lambda\) is empty, the library returns a setting that holds every decision (margin \(10^{12}\)) and exit code 2. With zero errors the p-value is \((1-\alpha)^{k}\), so certification is possible only once

\[ k \ \ge\ \frac{\ln(\delta / K)}{\ln(1 - \alpha)} \]

acted-on cases are in the risk part (minimum_trials_to_certify). For the fixed sequence, \(K\) is 1 in this formula (each test uses the full \(\delta\)): 59 for 5% at 95%, 149 for 2%. For Bonferroni over 1,200 settings: 197 for 5%, 500 for 2%. Under the fixed sequence, nothing is certified also when the first candidate in the order fails, whatever comes after it; the learner then warns that testing stopped at the first candidate.

Reporting: Clopper–Pearson bounds¶

The hold-out line and the RiskMonitor report one-sided Clopper–Pearson bounds. The upper bound at confidence \(c\) after \(e\) errors in \(k\) trials is the largest rate \(r\) still consistent with the data,

\[ u(e, k, c) = \sup\bigl\{ r : P(\mathrm{Bin}(k, r) \le e) > 1 - c \bigr\}, \qquad u(0, k, c) = 1 - (1 - c)^{1/k} . \]

For the hold-out: 1 error in 120 acts gives \(u = 3.89\%\) at 95%. Three reasons for this choice:

Exact at any size. It is computed from the binomial tail itself, so it never covers less than \(c\), even with 0, 1 or 2 errors. The familiar normal-approximation interval undercovers there, and with 0 errors it collapses to zero width.
It matches the test. \(u(e, k, 1 - \delta) \le \alpha\) exactly when the p-value above is at most \(\delta\), so the report and the certificate speak the same language. The reported bound is rounded outwards (below), so a reported \(u \le \alpha\) still implies \(p \le \delta\); the converse can fail only when \(\alpha\) is within one rounding step above the exact bound.
It is honest only where nothing was chosen. The hold-out bound is valid because the hold-out took no part in choosing. A bound computed on the risk part would use the data the choice was made from, so it would be optimistic; that is why explain() prints only the raw risk-part counts, labelled optimistic, and states the certificate (\(R \le \alpha\)) separately.

The monitor also uses the lower bound \(\ell(e, k, c) = \inf\{ r : P(\mathrm{Bin}(k, r) \ge e) > 1 - c \}\): when even \(\ell\) is above the target, the recent error rate is confidently too high.

Rounding. Both bounds are reported on a grid of decimals with 12 significant digits, rounded outwards: the upper bound is the smallest grid value \(g\) whose float passes the exact test \(P(\mathrm{Bin}(k, g) \le e) \le 1 - c\), and the lower bound the largest grid value with \(P(\mathrm{Bin}(k, g) \ge e) \le 1 - c\), both tails computed exactly in integers. So the reported upper bound is at or above \(u\) and the reported lower bound at or below \(\ell\) (never less conservative, by at most one part in \(10^{11}\)), and the value is the same on every platform, which the parameters' fingerprint relies on (tests/test_portability.py; design decision D69).

Assumptions¶

Independent cases from one process. The binomial test needs the risk-part cases to be independent draws from the same distribution as future cases (the code's docstrings say "exchangeable"; for this test, read it as independent and identically distributed). Cases that share related cases are not fully independent: decisions about ten accounts in the same fraud ring share their evidence and often their outcome. The test then treats ten correlated cases as ten independent ones and is overconfident. If your history has many such clusters, keep one decision per cluster in the risk part.
The pipeline is fixed. The certificate is about the whole pipeline as it ran in the history: your event source and model, how the graph was built, the policy and the checks. Blind spots of the checks (direction is checked only when support is measured; agreement can be inflated by an imbalanced label mix) appear as measured errors, not as hidden ones. Change any part and the certificate no longer applies.
The calibration scores are part of the setting. The scores are fixed by the calibration part and deployed as they are (LearnedParameters.apply). Refreshing them on another schedule changes the setting the certificate is about.

What it does not cover¶

Drift. If the process changes after the history was collected, nothing in the certificate says so. That is the job of RiskMonitor and of re-learning on a schedule.
Selective labels. If outcomes are known only for some decisions (only the cases a reviewer saw, only the cards that were blocked), the history is drawn from a different distribution from the one the service decides on. The certificate is then about that other population. The learner cannot detect this from the outcomes alone; when cases carry was_acted_on, it warns if every case with a known outcome was acted on, and when none carries it, it says it cannot tell.
Label delay. Outcomes that arrive late (chargebacks after 60 days) make recent history over-represent quickly resolved cases, which are often the easy ones.
Subgroups and single cases. \(R < \alpha\) is an average over all cases. A 5% rate overall can hide 20% in a small subgroup; the guarantee says nothing about any one decision.
Repeated learning. Each run has its own \(\delta\). Re-running with new seeds until a result looks good, and keeping that one, spends the confidence several times over.

Conservativeness, measured¶

At 90% confidence the guarantee allows up to 10% of learned settings to miss the target. The repository's test (tests/test_learn.py::test_guarantee_holds_across_independent_histories, run for both procedures) learns on 60 independent synthetic histories of 1,200 cases each, for a target of 8% at 90% confidence over a grid of 54 settings, and measures each chosen setting's true error rate and act rate on 40,000 fresh cases. Run from the repository root (uv run python ltt_check.py):

"""Does the certificate hold? Learn on 60 independent histories, test each on fresh cases."""

import sys

import numpy as np

sys.path.insert(0, "tests")  # run from the repository root
from test_learn import _synthetic_history

from grounded_decision.learn import (
    ParameterGrid,
    RiskTarget,
    case_arrays,
    learn_parameters,
    predicted_acts,
)

target = RiskTarget(max_risk=0.08, confidence=0.90)
grid = ParameterGrid(
    reconstruction_margins=(-1.0, 0.0, 0.5, 1.0, 2.0, 4.0),
    window_prefactors=(0.0, 1.0, 2.0),
    conformal_alphas=(0.0, 0.05, 0.2),
    minimum_supports=(0.0,),
)
fresh = _synthetic_history(40_000, np.random.default_rng(999))
for procedure in ("fixed_sequence", "bonferroni"):
    error_rates, act_rates = [], []
    for repetition in range(60):
        history = _synthetic_history(1200, np.random.default_rng(repetition))
        learned = learn_parameters(
            history, target, grid=grid, seed=repetition, procedure=procedure
        )
        if not learned.certified:
            continue
        arrays = case_arrays(fresh, learned.calibration_scores)
        acts = predicted_acts(
            arrays,
            margin=learned.reconstruction_margin,
            prefactor=learned.operational_fk_prefactor,
            decision_line=learned.decision_line,
            alpha=learned.conformal_alpha,
            minimum_support=learned.minimum_support,
        )
        act_rates.append(acts.mean())
        error_rates.append(
            np.count_nonzero(acts & arrays.wrong) / max(1, np.count_nonzero(acts))
        )
    error_rates = np.array(error_rates)
    print(
        f"{procedure}: certified {error_rates.size} of 60 histories (grid size {grid.size})"
    )
    print(f"  true error rate above 8%: {np.count_nonzero(error_rates > 0.08)}")
    print(
        f"  true error rate: mean {error_rates.mean():.1%}, max {error_rates.max():.1%}"
    )
    print(f"  share of fresh cases acted on: mean {np.mean(act_rates):.1%}")

fixed_sequence: certified 56 of 60 histories (grid size 54)
  true error rate above 8%: 1
  true error rate: mean 3.6%, max 8.8%
  share of fresh cases acted on: mean 26.9%
bonferroni: certified 51 of 60 histories (grid size 54)
  true error rate above 8%: 0
  true error rate: mean 2.5%, max 6.9%
  share of fresh cases acted on: mean 23.2%

Both procedures keep the guarantee: 1 miss in 56 certified settings for the fixed sequence and 0 in 51 for Bonferroni, against an allowed 10%. The fixed sequence certifies on more histories, sits closer to the target (mean true error 3.6% against 2.5%) and acts more often (26.9% of fresh cases against 23.2%). That is the trade it is chosen for: less room to spare, more automatic decisions, the same guarantee. tests/test_learn.py::test_fixed_sequence_acts_more_often_than_bonferroni checks the gain.

Where Bonferroni's room comes from:

It treats the settings as unrelated. They are not: a margin of 2.0 and one of 1.0 act on mostly the same cases. Dividing \(\delta\) by \(K\) pays for 54 (or 1,200) independent chances of a false "yes" when the real number is much smaller. The fixed sequence pays nothing for the grid size; it pays instead when an unpromising candidate sits early in the order.
The test is at the least favourable point. It guards against a true rate exactly at the target; most candidates are far from it.
The choice is coarse. The best certified setting on a grid is usually some distance inside the target.

How to make it less conservative.

Use a smaller grid, chosen in advance. Under Bonferroni, fewer candidates mean a smaller \(K\). Under the fixed sequence, dropping values you would never deploy keeps them out of the order. Either way, the guarantee holds for any grid fixed before seeing the risk data.
Holm's step-down procedure (not implemented) certifies everything Bonferroni does and sometimes more, with the same guarantee. Angelopoulos et al. also discuss graph-based variants of the fixed sequence.

Where the code lives, and what to read next¶

grounded_decision.learn.learn_parameters: the procedure (with the validity argument for both procedures in its docstring); predicted_acts reproduces exactly what Module.decide would do (checked case by case in tests/test_learn.py).
grounded_decision.learn.statistics: the binomial tail, Clopper–Pearson bounds and minimum_trials_to_certify.
Learn API reference.
Coverage for unfamiliar cases (conformal prediction): the unusual-case check whose alpha is one of the learned settings. Its guarantee (at most about \(\alpha\) of typical correct cases held) is a different statement from the error-rate certificate on this page.
A. N. Angelopoulos, S. Bates, E. J. Candès, M. I. Jordan, L. Lei. Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control. arXiv:2110.01052, 2021.