Operate and monitor¶

Once the service enforces decisions, its act rate will move: more cases held one morning, fewer the next. Some moves are the world changing and the service doing its job; others are a broken evidence job, a model that was rescaled, or a policy deployed with the wrong threshold, and they look the same if you only watch the act rate. This page says what to record for every decision, which metrics and dashboards separate those causes, what to alert on, and what to do when an alert fires.

Who this is for: the engineer or SRE on call for a decision service, and the data scientist who owns its evidence and calibration.

What you need: a service built on the reference service (Integrate) and the expected rates and review load from Estimate impact. The examples use the reference service's GET /metrics, GET /report, GET /health and its audit table.

What to record for every decision¶

The reference service already writes all of this; if you built your own service, copy the list.

Where	What	Why
audit table, one row each	the full record (`DecisionRecord.to_dict()`), the evidence (`EvidenceSnapshot.to_dict()`), entity id, idempotency key, policy version, state before and after, routing, failed checks	rebuild and replay any decision; answer "why was this held?" months later
one structured log line each	ids, verdict, routing, failed checks, `d_lambda_sq`, `p_hat`, `p_threshold`, `conformal_p`, latency, policy version	search and correlate; never log the event payload (it may hold personal data)
counters (`/metrics`)	decisions by verdict and routing; check failures by check; latency; idempotent replays; state conflicts	alerting
with every record	the learned parameters' fingerprint (`params@…`, from `LearnedParameters.fingerprint`)	trace any decision to the settings, history and risk target behind it
when the outcome arrives	whether the action was right, joined to the decision; feed it to a `RiskMonitor`	check the live error rate against the certified target; the next learning run's history

The health numbers come from DecisionRecord.health. The ones that explain most changes:

Field	Plain meaning	A sustained change usually means
`d_lambda_sq`	evidence strength; must exceed 1 + margin (1.05 by default)	falling to 0: empty graphs (evidence job broken); drifting down: the related cases agree less (real change, or a label source changed)
`in_operational_window`	the density sat on the knife-edge	the cut-off no longer fits today's data, or graphs got smaller
`p_hat` vs `p_threshold`	link density against your cut-off	a new pattern of linking (or a hub you did not cap)
`conformal_p`	how typical this case's score is against the calibration set	drifting down: the model or population shifted, or calibration is stale

The dashboards¶

1. Decisions by routing, over time. From grounded_decision_decisions_total{verdict, routing}. Plot execute, human_review, await_next_event and execute_degraded separately. await_next_event costs nobody anything (the policy is waiting); do not add it to the hold rate.

2. Check failures, over time. From grounded_decision_channel_failures_total{channel}. This is the panel that tells causes apart. The counter counts every failed check on every decision, including abstains that proposed nothing, so read it next to panel 1.

3. Health numbers. Median and 10th percentile of d_lambda_sq and conformal_p, and the share of decisions with in_operational_window, from the log lines or the audit table.

4. Review queue. Open reviews (GET /health → open_reviews), age of the oldest open review, and reviews resolved per hour, against reviewer capacity.

5. Rollout comparison. GET /report for the running policy version: naive acts, grounded acts, withheld, failures by check. It resets with every new policy version.

Here is what the reference service reports after six requests (a ring block, a ring pass, two thin-chain holds, an account with no evidence, and a repeated suspicious on a card already blocked). Save as ops.py and run uv run python ops.py:

import json
import logging
import tempfile
from pathlib import Path

from grounded_decision.examples.reference_service import (
    build_service,
    parse_decide_request,
)

logging.disable(logging.INFO)
service = build_service(Path(tempfile.mkdtemp()) / "ops.sqlite3", seed_demo=True)
traffic = [
    ("acct_000", "suspicious"),
    ("acct_005", "clean"),
    ("acct_900", "suspicious"),
    ("acct_902", "suspicious"),
    ("acct_777", "suspicious"),
    ("acct_000", "suspicious"),
]
for number, (account, event) in enumerate(traffic):
    body = {
        "entity_id": account,
        "event": {"name": event},
        "idempotency_key": f"auth-{number}",
        "candidate_model_score": 0.1,
    }
    service.decide(
        parse_decide_request(body, service.engine.valid_event_names), f"r{number}"
    )
print(service.metrics.render().rstrip())

# Hold rate by failed check, from the audit table: the query behind a dashboard panel.
with service.database.connection() as connection:
    for row in (
        connection.execute("""
            SELECT routing, failed_channels, COUNT(*) AS decisions,
                   ROUND(AVG(json_extract(record_json, '$.health.d_lambda_sq')), 2) AS strength,
                   ROUND(AVG(json_extract(record_json, '$.health.conformal_p')), 2) AS conformal_p
            FROM decisions GROUP BY routing, failed_channels ORDER BY decisions DESC""")
    ):
        print(dict(row))
print(json.dumps(service.health()["open_reviews"]))

# TYPE grounded_decision_decisions_total counter
grounded_decision_decisions_total{verdict="abstain",routing="await_next_event"} 1
grounded_decision_decisions_total{verdict="abstain",routing="human_review"} 3
grounded_decision_decisions_total{verdict="act",routing="execute"} 2
# TYPE grounded_decision_channel_failures_total counter
grounded_decision_channel_failures_total{channel="reconstruction"} 3
grounded_decision_channel_failures_total{channel="threshold"} 3
# TYPE grounded_decision_decision_latency_seconds summary
grounded_decision_decision_latency_seconds_sum 0.010279
grounded_decision_decision_latency_seconds_count 6
# TYPE grounded_decision_idempotent_replays_total counter
grounded_decision_idempotent_replays_total 0
# TYPE grounded_decision_state_conflicts_total counter
grounded_decision_state_conflicts_total 0
# TYPE grounded_decision_outcomes_recorded_total counter
grounded_decision_outcomes_recorded_total{action="correct"} 0
grounded_decision_outcomes_recorded_total{action="wrong"} 0
{'routing': 'human_review', 'failed_channels': '["reconstruction", "threshold"]', 'decisions': 3, 'strength': 0.11, 'conformal_p': 1.0}
{'routing': 'execute', 'failed_channels': '[]', 'decisions': 2, 'strength': 1.51, 'conformal_p': 1.0}
{'routing': 'await_next_event', 'failed_channels': '[]', 'decisions': 1, 'strength': 1.51, 'conformal_p': 1.0}
3

The SQL query is the shape of a dashboard panel: decisions grouped by routing and failed checks, with the average evidence strength and conformal p of each group. The three holds failed agreement and the knife-edge together, with average strength 0.11 (thin or empty graphs); the two acts had strength 1.51. Latency averages about 2 ms a decision, almost all of it reading evidence.

Alerts¶

Set thresholds from the range you measured in Estimate impact and confirmed in shadow mode, not from these examples.

Alert	Condition (example)	First suspect
Hold rate out of band	`human_review` share outside the replay's 90% range for 30 minutes	read panel 2 to see which check moved
Agreement failures jump with flat traffic	`channel_failures_total{channel="reconstruction"}` rate doubles; median `d_lambda_sq` near 0	evidence job, cache or key lookup returning empty graphs
Knife-edge failures jump	share with `in_operational_window` doubles	graphs got smaller, or density moved onto the cut-off
Unusual-case failures jump, or `conformal_p` drifts down	conformal failures rate doubles; median `conformal_p` falls week on week	model retrained or rescaled; calibration set stale or empty
Every decision abstains on `game` or `certificate`	any failure of these checks	a policy deployed with a structural problem; roll back
`409 policy_version_mismatch`	any	a deploy changed the policy or a setting without migrating saved states
Retry storm	`idempotent_replays_total` or `state_conflicts_total` spikes	a client retrying in a loop, or a hot entity
Queue backlog	open reviews above capacity, or oldest open review older than your SLA	holds above plan, or reviewers short
Pods disagree on policy	`GET /health` `policy_version` differs across instances	a partial deploy
Live error rate above the certified target	`RiskMonitor.status()` is `above_target`	the world changed since the history (new pattern, retrained model, new population); see the runbook

Runbook¶

Hold rate jumped.

Panel 2: which check's failures moved?
Reconstruction, with d_lambda_sq near 0: query a few recent held records and look at observations (accounts, links). Zero accounts means the evidence source is broken; fix it, and do not pad the graph to bring the rate back. Cases held meanwhile are in the review queue, which is the safe outcome.
Threshold: compare today's p_hat distribution with the one you chose the cut-off from. If the data moved, re-learn the settings on recent history (Learn parameters) and ship them through Roll out.
Conformal: compare today's candidate scores with the calibration set. A step change on a deploy day is a model change; re-learn with the new model's scores, which rebuilds the calibration set (Learn parameters).

Hold rate dropped, or nothing is held.

Has the calibration set gone empty? With require_conformal_calibration=False an empty set passes everything; with True it holds everything. Check metrics.n_cal in a recent record.
Did the candidate score stop varying (a default value sent by a caller)? Look at label_score across recent records.
Did someone move the cut-off far from the data or switch the threshold check off? Compare the policy version with the last approved one.

The error-rate monitor says above_target. The learned settings are certified for the process the history came from; a monitor that is confidently above the target says the process has changed.

Stop acting automatically: route every decision for this module to review (or deploy the fail-closed settings the learner produces when nothing is certified), while you investigate.
Look for the change: a model retrain or rescale, a new evidence source, a shift in who is being decided on. Compare recent label_score, d_lambda_sq and p_hat with the history.
Re-learn on recent labelled history (Learn parameters), check the hold-out line, and ship the new settings through Roll out.

One above_target can be a false alarm (the monitor is checked over and over), so investigate before concluding; but do not keep acting while you do.

409 policy_version_mismatch after a deploy. Pause writes, migrate saved states, resume (Roll out). If you cannot migrate yet, roll back the deploy.

Reviewers disagree with the holds. Pull resolved reviews with the failed checks. If most "held, reviewer says obviously fine" cases failed only the knife-edge check on small graphs, the window is doing what it does at small sizes (Troubleshoot); decide with the domain owner whether to widen neighbourhoods or accept the load.

Troubleshoot goes from a single surprising record to its cause.

Keep the loop closed¶

Review outcomes become evidence. In the reference service, resolving a review writes the reviewer's label into the outcomes table the evidence builder reads, so the next decision about any related entity sees it. Make sure your case tool calls the resolve endpoint.
Watch the live error rate. Feed each acted-on decision's outcome to a RiskMonitor built with the same RiskTarget you learned with:

from grounded_decision.learn import RiskMonitor, RiskTarget

monitor = RiskMonitor(RiskTarget(max_risk=0.05), window=500)
# 300 acted-on decisions with 1% wrong, like the certified history ...
for index in range(300):
    monitor.record(acted=True, action_was_correct=index % 100 != 0)
print(
    monitor.status().value,
    monitor.errors,
    monitor.trials,
    f"{monitor.upper_bound():.3f}",
)
# ... then the world changes: the next 300 are 20% wrong.
for index in range(300):
    monitor.record(acted=True, action_was_correct=index % 5 != 0)
print(
    monitor.status().value,
    monitor.errors,
    monitor.trials,
    f"{monitor.lower_bound():.3f}",
)

within_target 3 300 0.026
above_target 62 500 0.100

above_target means even the optimistic bound on the recent error rate (10.0% here) is above the 5% target. Export the status as a gauge and alert on it.

Re-learn on a schedule (monthly, or once per outcome cycle) from recent labelled history, and whenever the model or the evidence builder changes (Learn parameters). Re-learning rebuilds the calibration set per proposed action and re-certifies the settings together; refreshing the calibration set on its own changes what the certificate is about. In the reference service each new set of parameters changes the settings version (parameters_fingerprint) stamped on every decision; saved states stay valid.
Audit acted cases too. Reviews only produce outcomes for held cases. Sample a few acted cases a week for review, or you will never see the wrong acts (including acts in the wrong direction, which the checks do not catch), the monitor has nothing to count, and the next learning run inherits selective labels.
Re-run the impact replay quarterly on the latest history and compare with what production shows.

What can go wrong¶

Alerting on the act rate alone. A dead evidence job and a world on the knife-edge both lower it. Alert on the split.
Counting await_next_event as holds. It inflates the hold rate and the reviewer-load estimate.
Logging payloads. They can contain personal data; the audit table already has them under access control.
Treating decide_ms_budget as a timeout. It is a target the library does not enforce; enforce timeouts on the evidence query.

Next¶

Learn parameters from your history: re-learning when the monitor or the schedule says so.
Troubleshoot: from one record to its cause.
The decision record: every field you are storing.
Design-review checklist: what the alerts and runbook must cover before launch.