Adaptive loops: decisions on a stream of drifting evidence

A real service does not make one decision. It makes the same kind of decision every request, every hour or every alert, while the evidence behind it improves, decays, wanders around a cut-off or simply gets noisier. This track shows what the library does in each of those situations, and how to rehearse it before launch so you know how often the service will act on its own and how many cases will reach a person.

It is for data scientists sizing a launch, policy owners who want to know how often a decision will be held, and engineers building the dashboards. It assumes you have seen one decision (the step track or Your first decision). Terms are defined in the glossary.

uv run grounded-decision examples --track adaptive
uv run grounded-decision examples adaptive_evidence_ramp

The source is src/grounded_decision/examples/adaptive.py. For the task pages built on these runners, see Estimate impact before launch (act rate and review-queue size) and Operate and monitor (drift dashboards and alerts).

The two rehearsal tools: a baseline and a spread

To rehearse, you need to say what a typical day's evidence looks like and how much one day strays from it. The library has one object for each, in grounded_decision.integrate.

EvidencePrior is the baseline: the evidence you expect on a typical decision, before you look at today's graph. In a running service it is the mean of each logged evidence field over, say, the last 30 days.

EvidencePrior field	What it means in production
n	typical number of cases in the evidence graph
degree	mean links per case
lambda_mean	mean neighbour agreement (λ)
p_mean	mean link density
p_threshold	your cut-off density (a tuning constant, not averaged)
label_mean	typical candidate score
n_cal, cal_loc, cal_step	an evenly spaced stand-in for your calibration set

mean_d_lambda_sq() returns the evidence strength at the baseline, degree × lambda_mean². If it is not comfortably above 1.05, the service will rarely act even on a calm day.

Volatility is the spread: how far one decision's evidence typically strays from the baseline, as a standard deviation per field. Three presets ship:

Name	links σ	agreement σ	density σ	score σ	Stands for
CALM	0	0	0	0	every draw equals the baseline; deterministic sweeps
MILD	0.4	0.04	0.02	0.05	a stable service on an ordinary week
WILD	2.0	0.20	0.12	0.35	an outage, a holiday, a broken pipeline, an attack

draw_snapshot(prior, volatility, rng=...) draws one simulated snapshot. It clips draws to legal ranges (links to [0, 64], agreement and density to [0, 1], score to [0, 2]) and never adds noise to the calibration set: refreshing the calibration set is a product decision, not a volatility setting.

Estimating them from your own logs

estimate_prior_and_volatility(logged_evidence, decision_density_threshold=...) takes the snapshots you already log with every decision and returns (EvidencePrior, Volatility): the mean of each field and its sample standard deviation. It lives in the examples module grounded_decision.examples.adaptive, which is example code, not part of the supported API. Copy its twenty lines into your own code rather than importing it in production.

from grounded_decision.examples.adaptive import estimate_prior_and_volatility
from grounded_decision.integrate import DecisionSession, draw_snapshot
from grounded_decision.types import Event, EvidenceSnapshot
import numpy

# Last 30 days of decisions, read back from your decision log.
logged_evidence = [
    EvidenceSnapshot.from_dict(row["evidence"]) for row in load_decision_log(days=30)
]
baseline, day_to_day_spread = estimate_prior_and_volatility(
    logged_evidence, decision_density_threshold=0.08
)

random_generator = numpy.random.default_rng(0)
session = DecisionSession(build_my_policy(), reset_each_tick=True)
for _ in range(1000):
    session.step(
        Event("suspicious"),
        draw_snapshot(baseline, day_to_day_spread, rng=random_generator),
    )
automatic_rate = sum(record.acted for record in session.records) / 1000

automatic_rate is the share of cases the policy handles without a person on a typical day; one minus it, times your daily volume, is the review queue. Re-run with WILD to size the queue for a bad day. (load_decision_log and build_my_policy are yours.) Because draws are clipped, a spread estimated from simulated draws under WILD comes out a little smaller than the preset that produced them.

Sessions and trajectories

• DecisionSession(module, reset_each_tick=...) feeds a stream of (event, evidence) pairs, called ticks, to one module. reset_each_tick=True treats every tick as a new case (a new authorisation, post or alert). reset_each_tick=False keeps the policy's state between ticks (one ticket, one incident, one access request moving through a workflow).
• trajectory(records) turns a run into dashboard rows: t, event, verdict, d_lambda_sq, p_hat, in_operational_window, conformal_p, failed_channels. Every runner below returns it as summary["trajectory"].

The simulators (draw_snapshot, ramp_snapshots, random_walk_p) are for rehearsal in CI and design reviews. At decision time, always build the snapshot from live data with snapshot_from_graph.

The knife-edge check does not know which side of the line you are on

The knife-edge check refuses to decide when the link density is near your cut-off. Far from the line it passes on either side: a density far below the cut-off passes just as a density far above it does. If "above the line" and "below the line" mean different actions, your policy must receive different events for each side (for example dense_resharing and sparse_resharing) and propose a different action for each. Otherwise a density far below the line will pass the check and carry out the "above" action. See Write a policy and Are we on a knife-edge?.

adaptive_evidence_ramp

The situation. A newsroom wants to publish a short alert about a breaking claim automatically, once enough reports corroborate it. Publishing on the first report is how false alerts happen; waiting for an editor at 3am is how you end up an hour late. Each report is a case; two reports are linked when they describe the same scene and can be checked against each other; a report's label is whether it confirms the claim. The desk re-checks every ten minutes as reports arrive.

uv run grounded-decision examples adaptive_evidence_ramp

What to notice. Corroboration grows steadily from two loosely agreeing reports per report (evidence strength \( 2 \times 0.20^2 = 0.08 \)) to ten strongly agreeing ones (\( 10 \times 0.60^2 = 3.6 \)). The first six checks hold, with strength rising 0.08, 0.15, 0.26, 0.40, 0.59, 0.82, all below the 1.05 line. The seventh check, at minute 70 (first_publish_minutes_after_first_report = 70), reaches 1.11 and publishes. Density and scores never approach their limits, so neighbour agreement is the only check that moves.

The takeaway. The library waits for corroboration rather than for a fixed delay. In production, stop re-checking after the first publish, and build each check's snapshot from the reports collected so far.

adaptive_threshold_walk

The situation. A trust-and-safety desk takes down posts amplified by coordinated accounts. The rule "take down when the reshare density exceeds 0.20" flips every hour while the density hovers near 0.20, and every flip is a user-visible action and an appeal. The 200 accounts that shared the post are the cases; two are linked when one reshared the other within a minute. Every other piece of evidence is held strong.

uv run grounded-decision examples adaptive_threshold_walk

What to notice. The hourly density takes a random walk from 0.20 with an hourly standard deviation of 0.03. With 200 accounts the module refuses any density within 0.20 ± 0.057 (window_half_width = 0.057). The densities run 0.21, 0.235, 0.245, …, so the first eight hours are held. From hour 8 (density 0.259) the post is taken down; hour 14 dips back to 0.243 and is held again. In total, 9 hours held (hours_inside_threshold_window = 9) and 7 takedowns.

The takeaway. Near the cut-off the library refuses to flip a user-visible decision back and forth. Read the warning above: in this run the density stays above 0.20, but a policy that proposes a takedown on every hourly review would also take the post down at a density far below the line. Emit separate events for each side.

adaptive_volatility_regimes

The situation. The fraud dashboard shows the average evidence quality, and the average is healthy. How often will an individual flagged authorisation fall below the line anyway? The baseline is 8 links per account, agreement 0.55 (strength 2.42), density 0.35 against a 0.08 cut-off. The runner draws twenty flagged authorisations under each of CALM, MILD and WILD and decides each with the fraud_review policy.

uv run grounded-decision examples adaptive_volatility_regimes

What to notice. blocks_by_regime is {calm: 20, mild: 20, wild: 11}. On the wild day nine authorisations are held instead of auto-blocked: five fail neighbour agreement (strengths such as 0.17, 0.86 and 1.02, all at or below 1.05), four have a density inside the knife-edge window (one of them also fails agreement), and one fails the unusual-case check (p = 0.049). The run also feeds each regime's draws back through estimate_prior_and_volatility; for MILD it recovers 0.36 links, 0.035 agreement, 0.017 density and 0.055 score, close to the preset.

The takeaway. A healthy average hides a tail of individual cases below the line. Estimate your own spread from logs to see which preset your service lives in, and size the review queue from the bad-day run, not the average.

adaptive_prior_grid

The situation. Before building a pipeline that labels online communities automatically (for example "medical advice"), a team has to decide how many links per member to collect, given how strongly linked members' labels agree.

uv run grounded-decision examples adaptive_prior_grid

What to notice. The grid is links per member {2, 4, 8, 16} by agreement {0.20, 0.35, 0.55, 0.80}, one deterministic decision per cell. The module labels only where links × agreement² exceeds 1.05:

• 16 links at agreement 0.20 give 0.64: held;
• 8 links at agreement 0.35 give 0.98: held, just short;
• 4 links at agreement 0.55 give 1.21: labelled;
• 2 links need agreement 0.80 (1.28).

summary["grid_cells"] lists all sixteen cells.

The takeaway. Agreement matters more than volume, because it enters squared: doubling agreement is worth four times as many links. Run the same sweep with your own measured agreement to decide which data source to build first.

adaptive_workflow_over_time

The situation. The access-request workflow from the step track, now repeating: (test(access_requested) ; (test(manager_approved) ; act(grant_access) | test(manager_rejected) ; act(close_request)))*. The module is never reset. One engineer makes two requests a quarter apart: the first in the team's first week, when the peer group is barely linked (2 links each, agreement 0.2, density 0.10 near the 0.08 cut-off), the second once the team has settled.

uv run grounded-decision examples adaptive_workflow_over_time

What to notice.

Tick	Event	Evidence	Verdict	Proposed action	State
0	access_requested	new team	abstain	none (waiting)	s1
1	manager_approved	new team	abstain	grant_access, blocked	s3
2	access_requested	settled	abstain	none (waiting)	s1
3	manager_approved	settled	act	grant_access	s3

Tick 1 is a real hold: the approval cannot be backed by peer evidence, so a person decides. Ticks 0 and 2 are waiting for approval; on tick 0 the data checks also fail, but nothing was proposed, so nothing was blocked.

The takeaway. Where a request is in the workflow and how good its evidence is are independent. A request can be at the right step with the wrong evidence (tick 1), or have good evidence and nothing to do yet (tick 2). Log both extras["state"] and the snapshot.

adaptive_incident_loop

The situation. On-call automation that mitigates and pages, using the policy from the incident response case: alert starts mitigation, still_bad pages, ack returns to mitigation, clear resolves, and a new alert reopens. Hosts are the cases, linked when they share a dependency (database, cache, load balancer); a host's label is whether it is alerting. One incident, five events, state kept throughout. Two of the events arrive with flappy evidence: one host whose neighbours disagree with it.

uv run grounded-decision examples adaptive_incident_loop

What to notice.

Event	Evidence	Verdict	Proposed	State after
alert	outage	act	mitigate	mitigating
still_bad	flappy	abstain	page_oncall	paging
ack	outage	act	mitigate	mitigating
clear	outage	act	close	resolved
alert	flappy	abstain	mitigate	mitigating

The flappy still_bad pages nobody, yet the saved state moves to paging. The flappy final alert does not reopen the incident's mitigation, yet the state moves to mitigating.

The takeaway. The state follows the events; the verdict only decides whether the action is carried out. After a hold, the saved state describes where the incident would be if the proposal had run. Your hold handler should treat the person's response as the next event (for example ack once they take over) so the state and reality line up again. The policy's progress check passes here, but that check is weak: it only fails for policies whose every reachable state is marked bad, and it does not detect general livelock (see Can the workflow keep going?). The holds on individual ticks come from the evidence.

adaptive_conformal_drift

The situation. A lender approves small credit lines automatically. The calibration set is the model's scores on last quarter's loans with known outcomes. A new acquisition channel then sends applicants unlike last quarter's, and the typical score climbs 0.12 per week. The graph evidence (applicants linked by shared employer and address history) stays healthy.

uv run grounded-decision examples adaptive_conformal_drift

What to notice. Weeks are numbered from 0. The unusualness p-value falls week by week: 1.0, 1.0, 1.0, 0.98, 0.83, 0.68, 0.54, 0.39, 0.24. In week 8 the score still has 9 of 40 calibration scores at or above it (p = 10/41 = 0.24) and is approved. In week 9 only 3 are (p = 4/41 = 0.098, below 0.10), and applications are held for an underwriter (first_week_held_for_underwriter = 9).

The takeaway. The unusual-case check is the drift alarm. The first hold is the signal to refresh the calibration set, and probably the model, on the new population.

adaptive_autoscaling_loop

The situation. A naive autoscaler scales out on every CPU alert, so one noisy instance can buy capacity all day. Instances are the cases, linked when they sit behind the same load balancer; an instance's label is whether it is saturated. Load ramps from a quiet fleet to sustained load over eight minutes and back over eight more, with MILD noise and a CPU alert every minute.

uv run grounded-decision examples adaptive_autoscaling_loop

What to notice. The module holds for the first four and last four minutes (evidence strength 0.03 to 0.95, below the line) and scales out in the middle eight (1.39 up to 4.88): scale_out_count = 8, where the naive autoscaler scales 16 times.

The takeaway. Scale-out follows fleet-wide saturation, not a single alert. In production, add a cool-down after each scale-out and a bound on consecutive holds while load stays high; the example leaves both out to show the raw verdicts. The cloud autoscaling case discusses the bound.

Independent ticks versus kept state

Six runners treat each tick as a new case: adaptive_evidence_ramp, adaptive_threshold_walk, adaptive_volatility_regimes, adaptive_prior_grid, adaptive_conformal_drift and adaptive_autoscaling_loop. They answer "what does this evidence do to the decision?"

adaptive_workflow_over_time and adaptive_incident_loop keep state. They answer "what does this sequence of events, plus evidence, do?" A flappy still_bad in the middle of an incident is a different hold from a flappy alert on a quiet system, because the policy is in a different place.

Mixing the two by accident is a common bug. If you keep state on a policy whose only path is "intake, then proposal", the second tick arrives at a state with no matching transition, and you get an abstain with every check passing and nothing proposed. step_unknown_event_stutter on the step track shows that in one record.

Wiring this into your system

1. Log every snapshot. Store evidence.to_dict() with each record.to_dict(). Without it you cannot estimate a baseline or explain a hold later.
2. Estimate nightly. Recompute the baseline and spread over the last 30 days. Alert when the baseline evidence strength (prior.mean_d_lambda_sq()) drifts toward 1.05: the service is about to stop acting even on ordinary days.
3. Rehearse in CI. For each policy change, draw a few hundred snapshots from the estimated baseline under the estimated spread and under WILD, and assert that the automatic rate stays within the range your team agreed. Fix the seed (numpy.random.default_rng(0)) so counts are reproducible.
4. Serve on live data only. At decision time, build the snapshot with snapshot_from_graph; never decide on a simulated draw.
5. Choose the session shape. Reset per tick when each tick is a new case; persist and restore extras["state"] when ticks belong to one entity (see step_json_roundtrip).
6. Dashboard the trajectory. Plot d_lambda_sq against 1.05, in_operational_window, and conformal_p against 0.10 over time. failed_channels tells you which check is behind a rise in holds.

A tick that fails two checks is still one abstain, with a longer reason. The module does not produce a severity score; log the list of failed checks instead. Nothing inside the module learns a baseline either. If you want a Bayesian update, compute it outside the module, put the result into an EvidenceSnapshot, and decide.