Skip to content

Where every number comes from

A decision record is full of numbers: 40 related accounts, agreement 0.85, a bar at 1.05, an alpha of 0.10. If nobody can say where a number came from, nobody can defend the decision it produced. This page sorts every number the library uses into five kinds (measured, learned, business input, domain definition, engineering budget), says how each is obtained, and how an auditor can check it.

It assumes you have seen one decision (Your first decision); terms are in the glossary.

Try it in your browser

The showcase site's the parameter-learning page shows the provenance record of every learned setting (the dataset and parameter fingerprints, the certificate, explain()) as the learner computes it in your browser. It runs this package itself, in Pyodide.

The shown numbers are the documented example, as the engine computed it; change a setting to compute it in your browser.

The parameter learner runs here when JavaScript is on; the parameter-learning page has it too.

The question

Picture a fraud reviewer asking why a card was blocked. "Evidence strength 4.3 was above the bar of 3" answers half of it. The other half is: who set the bar at 3, and why not 1.05 or 10? If the honest answer is "it was in an example" or "someone tuned it until the queue looked right", the service has an error rate nobody can state.

The library's answer is that only a few numbers need a person, and each of those has a plain meaning. The rest are either measured from your data on every decision, or learned from your labelled history with a certified error rate and a record of how.

The five kinds of number

Kind Who or what sets it Example How to audit it
Measured Computed from your graph of related cases, per decision degree, lambda_hat Rebuild the evidence from the stored graph and compare.
Learned Chosen from labelled history by Learn-then-Test reconstruction_margin, calibration scores Re-run the learner on the same history and seed; the fingerprint must match.
Business input The person who owns the risk "at most 5% of automatic blocks wrong, 95% sure" It is written in the design review and in the provenance record.
Domain definition The domain owner, with the data team "two accounts are related if they used the same device" Read the evidence builder's code and the policy.
Engineering budget Whoever runs the service max_dfa_states = 256 They bound cost, not decisions on data; check they are not hit.

Every evidence field

These are the fields of EvidenceSnapshot, built for each decision.

Field Kind How it is obtained How to audit it
n measured Number of related cases in the graph your builder drew (snapshot_from_graph counts the labels). Count the related cases stored with the record.
p_hat measured Link density: links ÷ \(n(n-1)/2\). Recount from the stored links.
degree measured Average links per case: \(2 \times\) links ÷ \(n\). Recount from the stored links.
lambda_hat measured Neighbour agreement: how much more often linked cases share a label than chance. Consider agreement_baseline="label_mix" when one label dominates everywhere and links may be random (below). Recompute with snapshot_from_graph on the stored labels and links.
support measured Share of related cases carrying the outcome the proposed action implies; snapshot_from_graph(..., supporting_label=...) computes it. None when not measured. Recount from the stored labels and the action-to-label mapping.
n_states domain definition The number of label values you defined (2 for fraud / cleared). Compare with the label definition in the builder.
label_score measured Your model's unusualness score for this case, computed the same way as in history (for example 1 − the model's probability that the proposed action is right). Recompute from the model version recorded with the decision.
scores learned The calibration set: label_score of past cases where the same proposed action turned out right, taken from the learner's calibration split. LearnedParameters.apply fills it in per action. They are listed in params.json; the fingerprint on the record names the file.
p_threshold domain definition, or learned The link density at which your decision would flip. Set it from the domain ("denser than 0.08 looks like a ring"), or let the learner choose it from quantiles of observed density with --learn-decision-line. The design review, or decision_line in params.json.
observations yours Notes for your audit trail. No check reads them. Not needed for the decision.

When most related cases share one label everywhere (most accounts are clean) and links may mean nothing, those links still agree most of the time, and the default agreement measure reports it as signal. With 60 accounts, 10% fraud and 146 random links (tests/test_evidence.py), the default gives \(\lambda = 0.493\) and evidence strength 1.18, which passes; with agreement_baseline="label_mix", \(\lambda = -0.145\) and strength 0.10, which holds. The default stays uniform because it is the measure the Kesten–Stigum result assumes, and because label_mix misjudges a homogeneous ring (nearly all fraud: strong evidence, little variation inside it, so it scores near zero). Which way the related cases point is the support check's job; the error rate of the whole pipeline is what learning certifies. Do the related cases agree? explains the difference.

Every setting

These are the fields of Limits, the learner's target, and the inputs that sit outside both.

Setting Kind Default How it is obtained
reconstruction_margin learned 0.05 Learn-then-Test, from the risk split. Candidates include -1.0 (check effectively off).
operational_fk_prefactor learned 2.0 Learn-then-Test. Sets the knife-edge window width; 0.0 switches the window off.
conformal_alpha learned 0.10 Learn-then-Test. 0.0 switches the unusual-case check off.
min_support learned 0.5 Learn-then-Test (minimum_support in params.json). 0.0 switches the support check off.
require_support engineering choice False Set True once every decision's evidence measures support, so a missing share holds instead of passing.
p_threshold (decision line) domain, or learned none See the table above.
require_conformal_calibration engineering choice False Set True in production so an empty calibration set holds instead of passing.
friedgut_kalai_prefactor published constant \(2^{19}\) The theorem's constant. Reported only; never changes a verdict.
ε in the knife-edge width fixed 0.1 Fixed in the decision path; the learned prefactor scales the whole width, so ε needs no separate setting.
max_risk, confidence, measure (RiskTarget) business input none The error rate you accept, how sure the certificate must be, and whether the rate is among acts or among all decisions. Data cannot choose your risk appetite.
What a related case is, what a label is, what "the action was correct" means domain definition none You decide. The learner measures how well your definitions work: a poor definition shows up as nothing certified or a low act rate.
The policy (events, states, actions) domain definition none You write it (Write a policy).
max_star_height, max_dfa_states, max_nfa_states, max_game_vertices, max_parity_depth, max_karp_vertices, max_wl_k, max_table_cells_exact, max_switch_steps engineering budget see Defaults Cost caps, all enforced. They stop a policy or side question that would take too long; they are unrelated to your data.
decide_ms_budget, compile_ms_budget engineering budget see Defaults Targets checked on the library by tests/test_performance_budget.py, not timers in your service.

The learner's provenance record stores, for every learned value, the target, the grid of candidates, how many were certified, the split sizes and seed, a fingerprint of the history, and the hold-out result.

Why the defaults are only starting points

The defaults (margin 0.05, prefactor 2.0, alpha 0.10, support minimum 0.5) make the examples run and are a reasonable first guess. They are not tuned to any data, and they carry no error rate. On a synthetic history whose true error rates are known (tests/test_learn.py), the defaults acted on 26.2% of 40,000 fresh cases with 7.94% of those acts wrong, which would break a 5% target. Settings learned from 3,000 cases of the same history for a 5% target acted on 18.2% with 0.93% wrong. Run from the repository root (uv run python compare.py):

"""Hand-set defaults versus learned settings, judged on 40,000 fresh cases."""

import sys

import numpy as np

sys.path.insert(0, "tests")  # run from the repository root
from test_learn import _synthetic_history

from grounded_decision import DEFAULT_LIMITS
from grounded_decision.learn import (
    RiskTarget,
    case_arrays,
    learn_parameters,
    predicted_acts,
)

history = _synthetic_history(3000, np.random.default_rng(1))
fresh = _synthetic_history(40_000, np.random.default_rng(999))
learned = learn_parameters(history, RiskTarget(max_risk=0.05), seed=1)
arrays = case_arrays(fresh, learned.calibration_scores)
for name, margin, prefactor, alpha, support in (
    (
        "defaults",
        DEFAULT_LIMITS.reconstruction_margin,
        DEFAULT_LIMITS.operational_fk_prefactor,
        DEFAULT_LIMITS.conformal_alpha,
        DEFAULT_LIMITS.min_support,
    ),
    (
        "learned ",
        learned.reconstruction_margin,
        learned.operational_fk_prefactor,
        learned.conformal_alpha,
        learned.minimum_support,
    ),
):
    acts = predicted_acts(
        arrays,
        margin=margin,
        prefactor=prefactor,
        decision_line=None,
        alpha=alpha,
        minimum_support=support,
    )
    wrong = np.count_nonzero(acts & arrays.wrong) / np.count_nonzero(acts)
    print(
        f"{name}: margin {margin:g}, prefactor {prefactor:g}, alpha {alpha:g},"
        f" support {support:g} -> acts on {acts.mean():.1%}, {wrong:.2%} of acts wrong"
    )

defaults: margin 0.05, prefactor 2, alpha 0.1, support 0.5 -> acts on 26.2%, 7.94% of acts wrong
learned : margin 2, prefactor 0, alpha 0.02, support 0 -> acts on 18.2%, 0.93% of acts wrong

(Both rows use the learned calibration scores, so only the four settings differ. This synthetic history does not measure support, so the support minimum has no effect in either row.) On this data the learner raised the evidence bar from 1.05 to 3, switched the knife-edge window off and made the unusual-case check stricter (alpha 0.02): the window did not separate right from wrong decisions once the bar was high. On your data it may choose the opposite. That is the point: the data decides, within the risk you set.

The loop

Learning is not a one-off. The settings are certified for the process your history came from; outcomes from production feed the next round.

flowchart LR
    H[("Labelled history<br/>evidence, proposed action,<br/>was it right?")] --> L["Learn<br/>Learn-then-Test<br/>against your risk target"]
    T["Risk target<br/>(business input)"] --> L
    L --> S["Settings + provenance<br/>params.json, fingerprint"]
    S --> D["Decide<br/>module with learned Limits;<br/>fingerprint on every record"]
    D --> O["Outcomes<br/>reviews, chargebacks,<br/>follow-ups"]
    O --> M{"RiskMonitor"}
    M -->|within target| D
    M -->|above target, or on schedule| H

The steps, commands and outputs are in Learn parameters from your history.

What this does not settle

  • The human inputs stay human. The learner cannot tell you how much risk to accept, or whether "shared a device" is the right notion of a related account. It tells you how well your definitions work: if nothing is certified, or the act rate is tiny, revisit them.
  • A certificate is about the past process. It holds for new cases drawn from the same process as the history. A new fraud pattern, a retrained model or a change in who gets reviewed breaks that; the monitor and scheduled re-learning are how you notice.
  • Selective labels. If outcomes are known only for some decisions (only reviewed holds, only acted cards), the learned numbers describe those decisions, not all of them.
  • Measured values are only as good as the graph. Wrong links or wrong labels give wrong measurements, faithfully.

Where next