Skip to content

Build a decision service

Calling module.decide(event, evidence) in a Python prompt is one thing; a service your card authoriser can call, that remembers each card, survives retries, keeps an audit trail and sends held cases to a person, is another. This tutorial walks through a runnable HTTP service that ships with the package and does all of that, one concern at a time. Each step names the production problem, shows the code that solves it, and gives a curl command with output copied from a real run.

You should have done Your first decision and From your data to evidence. You need a development checkout (Install), curl, and optionally jq to shorten the output. Allow about an hour.

The service is src/grounded_decision/examples/reference_service.py. It uses only the Python standard library (http.server, sqlite3, json, logging, threading), so it adds no dependencies. It is meant to be copied: run it, read it, then replace one piece at a time with what your platform already has (Postgres for SQLite, FastAPI for http.server, your warehouse for the demo tables, your case tool for the review table).

Try it in your browser

The showcase site's the decision runtime page steps through this tutorial's four requests in your browser (the same policy, evidence, checks and routing as POST /decide) and lets you send more. It runs this package itself, in Pyodide.

The shown numbers are the documented example, as the engine computed it; change a setting to compute it in your browser.

The decision runtime runs here when JavaScript is on; the decision runtime page has it too.

The policy

The domain is card fraud review, from src/grounded_decision/examples/fraud_review.py. Each card starts in intake. A suspicious authorisation moves it to review, which proposes block; a clean one moves it to auto, which proposes pass. A hold event from either moves it to hold, which proposes nothing.

stateDiagram-v2
    [*] --> intake
    intake --> auto: clean
    intake --> review: suspicious
    auto --> hold: hold
    review --> hold: hold
    note right of auto: proposes pass
    note right of review: proposes block

There is no way back to intake and no suspicious transition out of review. Step 5 shows what that means for a second authorisation on the same card.

The evidence is the device graph around the flagged account: accounts that logged in from the same phone or laptop are linked, and each carries the outcome of its last fraud review, exactly as in From your data to evidence.

What you will have at the end

caller (card authoriser, queue consumer, batch job)
    |  POST /decide {entity_id, event, idempotency_key, candidate_model_score}
    v
DecisionRequestHandler      thin HTTP adapter: reads and writes bytes
    v
DecisionRoutes              paths to methods, errors to JSON (any server can call it)
    v
DecisionService.decide
    1. entity lock                       EntityLocks (in-process)
    2. idempotency lookup                IdempotencyStore
    3. load state + check policy version StateStore
    4. build evidence                    EvidenceProvider  <- your data plugs in here
    5. decide                            DecisionEngine (built once, locked)
    6. route                             route_decision (pure function)
    7. one transaction:                  state (optimistic version)
                                         + audit row (append-only)
                                         + review row or outbox row
                                         + idempotency key and response
    8. one JSON log line + metrics

Eleven endpoints: POST /decide, GET /decisions/<entity_id>, GET /reviews, POST /reviews/<id>/resolve, GET /outbox, GET /report, GET /metrics and GET /health, plus three for learning the settings from your own outcomes (step 11): POST /outcomes, GET /history.jsonl and GET /parameters.

Every request carries a candidate_model_score: your risk model's score for this authorisation. The service passes it to the evidence as the case's unusualness score (label_score), and the unusual-case check compares it with the scores of the last 40 confirmed fraud cases (0.40 to 1.18). It holds when the score is higher than almost all of them. Every request on this page sends 0.1, below all 40, so that check passes throughout and the other checks do the work.

Step 0: run it

make service

That runs uv run python -m grounded_decision.examples.reference_service --seed-demo-data. --seed-demo-data loads two groups of related accounts into the evidence tables:

  • a dense ring, acct_000 to acct_015: sixteen accounts cycling through eight phones, thirteen confirmed fraud;
  • a thin chain, acct_900 to acct_903: four accounts linked through three shared laptops.

Seeding is idempotent, so restarting is safe. Other flags: --host, --port (default 8080), --database (default grounded_decision_reference_service.sqlite3 in your temp directory), --shadow (step 10), --parameters, --allow-uncertified, --agreement-baseline and --seed-learning-demo (step 11) and --log-level DEBUG (adds the access log). Pass flags through SERVICE_ARGS, for example make service SERVICE_ARGS="--seed-demo-data --port 9000".

The process logs two lines and waits:

2026-10-09 12:56:17,637 serving http://127.0.0.1:8080 policy=fraud_review@557323c7e95c settings=settings@13447e40df90 learned=none mode=enforce database=/var/folders/.../grounded_decision_reference_service.sqlite3
2026-10-09 12:56:17,637 try: curl -s http://127.0.0.1:8080/decide -d '{"entity_id": "acct_000", "event": {"name": "suspicious"}, "idempotency_key": "try-1", "candidate_model_score": 0.1}'

In a second terminal, check that it is alive:

curl -s localhost:8080/health

{
  "database": "/var/folders/.../grounded_decision_reference_service.sqlite3",
  "enforcement": "enforce",
  "enforcement_reason": "enforcing",
  "learned_parameters": null,
  "open_reviews": 0,
  "parameters_fingerprint": "settings@13447e40df90",
  "policy_name": "fraud_review",
  "policy_version": "fraud_review@557323c7e95c",
  "risk_monitor": {
    "detail": "start with --parameters to monitor the error rate",
    "status": "not_configured"
  },
  "status": "ok",
  "valid_events": [
    "clean",
    "hold",
    "suspicious"
  ]
}

Now make one decision: a suspicious $840 electronics purchase on a card inside the dense ring.

curl -s localhost:8080/decide -d '{
  "entity_id": "acct_000",
  "event": {"name": "suspicious", "payload": {"amount_usd": 840, "merchant": "electronics"}},
  "idempotency_key": "auth-0001",
  "candidate_model_score": 0.1
}' | jq '{routing, verdict: .record.verdict, action: .record.action,
          reason: .record.reason, state_before, state_after,
          evidence_summary, outbox_id, review_id, policy_version, parameters_fingerprint}'

{
  "routing": "execute",
  "verdict": "act",
  "action": "block",
  "reason": "all channels passed",
  "state_before": null,
  "state_after": "review",
  "evidence_summary": {
    "accounts": 16,
    "shared_device_links": 40,
    "truncated": false
  },
  "outbox_id": 1,
  "review_id": null,
  "policy_version": "fraud_review@557323c7e95c",
  "parameters_fingerprint": "settings@13447e40df90"
}

The card is blocked. Without the jq filter the response also has request_id, decision_id, entity_id, idempotency_key, enforced, and under record the full decision record: all six checks with their numbers, the health summary and extras.state (The decision record). The rest of this page explains how each field got there.

Step 1: the policy, built once and versioned

The concern. Building a Module runs the policy checks (star height, and the game that asks whether the workflow can keep going). That is the expensive part; decide itself takes a few microseconds for this policy (about 7 µs on a laptop). So the module is built once at start-up. And because a saved state such as "review" is a position in one version of the policy, every saved state and every audit row must say which version produced it.

The code. build_fraud_engine builds the module and fingerprints it:

def build_fraud_engine(evidence_parameters: Mapping[str, object]) -> DecisionEngine:
    module = fraud_review.build()
    return DecisionEngine(module, policy_fingerprint(module, evidence_parameters))

The engine carries two versions, because two different things change:

  • policy_fingerprint hashes what gives a saved state its meaning: every state with its verdict, action, payoff and priority; every transition; the initial state; the envelope; and the policy text if there is one. The result is a short string such as fraud_review@557323c7e95c. It is stored with each card's state, stamped on each audit row, and checked when state is restored (step 5).
  • parameters_fingerprint hashes the settings that change what the checks conclude: the Limits (margins, knife-edge window, unusualness level), the evidence settings (the ring density line, the calibration scores, the neighbourhood caps, the agreement baseline) and, when loaded, the learned parameters (step 11). The result looks like settings@13447e40df90. It is stamped on every audit row and returned with every decision, but it never blocks a saved state: a new calibration set or a re-learned margin does not change what review means, so cards keep their state across it.

Two processes running the same code get the same strings, so a rolling deploy of unchanged code is seamless.

Try it. GET /health (above) shows the running version, and valid_events lists the event names the policy understands. Request validation checks against that list (step 3).

Step 2: the evidence provider, where your data plugs in

The concern. In production nobody types evidence numbers in. You build the graph of related cases from raw rows and let snapshot_from_graph compute the numbers. The question is where that code lives and how you swap the demo tables for your warehouse.

The code. One interface is the seam:

class EvidenceProvider(Protocol):
    def evidence_parameters(self) -> Mapping[str, object]:
        """Settings that change decisions (thresholds, calibration, caps)."""
        ...

    def evidence_for(
        self, entity_id: str, candidate_model_score: float
    ) -> EvidenceSnapshot:
        """Return the evidence for ``entity_id`` right now."""
        ...

The shipped implementation, SqliteDeviceGraphEvidenceProvider, reads two tables, device_logins(account_id, device_id, logged_in_at) and review_outcomes(account_id, outcome, reviewed_at, source). For one account it walks outwards: the account, its devices, the other accounts on those devices, their devices, and so on. It stops at 500 accounts and skips any device used by more than 50 accounts (a shop kiosk or an office network is not evidence of a ring). It labels each account with its most recent review outcome (never-reviewed accounts count as cleared) and passes the rows to fraud_review.device_graph_evidence, which calls snapshot_from_graph:

def evidence_for(self, entity_id, candidate_model_score):
    with self.database.connection() as connection:
        device_logins, truncated = self._neighbourhood_logins(connection, entity_id)
        accounts = sorted({account for account, _ in device_logins})
        outcomes = self._last_outcomes(connection, accounts)
    snapshot = fraud_review.device_graph_evidence(
        device_logins, outcomes, candidate_model_score=candidate_model_score
    )
    return dataclasses.replace(
        snapshot,
        observations={
            **snapshot.observations,
            "entity_id": entity_id,
            "truncated": truncated,
        },
    )

The ring account sees 16 accounts and 40 shared-device links. The chain account sees 4 accounts and 3 links. That is the whole difference between the two decisions in step 4.

Swapping in your warehouse. Keep the walk and replace the three small query methods (_devices_of, _accounts_on, _last_outcomes) with queries against your tables. If a walk per request is too slow, run it in a batch job every few minutes, cache EvidenceSnapshot.to_dict() per account, and have evidence_for read the cache and set label_score from the request. Three rules hold whatever the source:

  • Missing data must produce thin evidence, never padded evidence. An account with no logins gets an empty graph and the decision is held. That is the right answer.
  • Bound the work: cap the neighbourhood and set a query timeout.
  • Return every setting that changes decisions from evidence_parameters, so changing a threshold changes the settings version and you can tell which decisions it made.

Build evidence from your data covers choosing the graph, the labels and the calibration set for other domains.

Try it. Decide for an account the evidence tables have never seen:

curl -s localhost:8080/decide -d '{"entity_id": "acct_777",
  "event": {"name": "suspicious"}, "idempotency_key": "auth-0002",
  "candidate_model_score": 0.1}' \
  | jq '{routing, verdict: .record.verdict, reason: .record.reason, evidence_summary}'

{
  "routing": "human_review",
  "verdict": "abstain",
  "reason": "d λ² = 0.0000 vs 1+0.050 (at or below KS; abstain); evidence density inside operational window (theorem width=1, vacuous=True)",
  "evidence_summary": {
    "accounts": 0,
    "shared_device_links": 0,
    "truncated": false
  }
}

No related cases, no automatic block. The card goes to a person.

Step 3: the request, validated at the edge

The concern. A malformed request must fail before any state is touched, and it must list everything that is wrong at once, so the caller fixes it in one round trip. Unknown event names need special care: the library treats an event the policy doesn't know as "stay put, propose nothing", which is safe but hides typos.

The code. parse_decide_request checks every field and collects every problem into one RequestValidationError (HTTP 400). The event name must be one of the policy's events. The handler also refuses bodies over 64 KiB and bodies without a Content-Length header.

Try it.

curl -s -w '\nHTTP %{http_code}\n' localhost:8080/decide \
  -d '{"entity_id": "acct_000", "event": {"name": "flagged"}, "candidate_model_score": "high"}'

{
  "error": "invalid_request",
  "message": "The request body is not a valid decide request: 'idempotency_key' must be a non-empty string; 'event.name' must be one of ['clean', 'hold', 'suspicious'], got 'flagged'; 'candidate_model_score' must be a finite number",
  "problems": [
    "'idempotency_key' must be a non-empty string",
    "'event.name' must be one of ['clean', 'hold', 'suspicious'], got 'flagged'",
    "'candidate_model_score' must be a finite number"
  ]
}

HTTP 400

Broken JSON gets the same treatment ("the body is not valid JSON (Expecting value: ...)"), and an unknown path gets a 404 that lists the paths that exist.

Step 4: decide and route

The concern. The library returns a verdict; your system needs to know which component handles it next. That mapping is the only business logic between the library and your side effects, so it should be small, pure and tested on its own.

The code.

def route_decision(verdict: Verdict, proposed_action: str | None) -> Routing:
    if verdict is Verdict.ACT:
        return Routing.EXECUTE
    if verdict is Verdict.DEGRADE:
        return Routing.EXECUTE_DEGRADED
    if proposed_action:
        return Routing.HUMAN_REVIEW
    return Routing.AWAIT_NEXT_EVENT
Verdict proposed_action Routing What the service does
act set execute writes the action to the outbox (step 7)
degrade any execute_degraded writes the policy's fallback action to the outbox
abstain set human_review opens a review with the proposal and the reason
abstain none await_next_event nothing: the policy is waiting for its next event

A proposed_action on an abstain is what the policy wanted. It is never permission to act. tests/test_reference_service.py tests every row of this table. Act, abstain, degrade explains the verdicts.

Try it. The ring card acted and routed to execute. Now the same $840 purchase on the chain account:

curl -s localhost:8080/decide -d '{"entity_id": "acct_900",
  "event": {"name": "suspicious", "payload": {"amount_usd": 840, "merchant": "electronics"}},
  "idempotency_key": "auth-0003", "candidate_model_score": 0.1}' \
  | jq '{routing, verdict: .record.verdict, action: .record.action,
         proposed_action: .record.proposed_action, reason: .record.reason,
         state_after, evidence_summary, review_id, outbox_id}'

{
  "routing": "human_review",
  "verdict": "abstain",
  "action": null,
  "proposed_action": "block",
  "reason": "d λ² = 0.1667 vs 1+0.050 (at or below KS; abstain); evidence density inside operational window (theorem width=4.39e+05, vacuous=True)",
  "state_after": "review",
  "evidence_summary": {
    "accounts": 4,
    "shared_device_links": 3,
    "truncated": false
  },
  "review_id": 2,
  "outbox_id": null
}

Same event, same model score, same policy:

  • The ring has 5 links per account on average and neighbour agreement 0.55, so evidence strength is \(5 \times 0.55^2 = 1.51\), above the 1.05 the check requires. Its link density is 0.33; with 16 accounts the knife-edge window reaches 0.209 either side of the line at 0.08 (up to 0.289), so 0.33 is clear of it.
  • The chain has 1.5 links per account and agreement 1/3 (two of its three links join accounts with the same outcome), so strength is \(1.5 \times (1/3)^2 = 0.17\). With only 4 accounts the window is as wide as it gets, 0.5 either side of the line, so its density of 0.5 is inside it. With 6 or fewer related accounts, the knife-edge check holds almost every decision.

A score cut-off would have frozen both cards. This service freezes one and sends the other to a person, as review 2 (review 1 is acct_777 from step 2).

Step 5: state that survives between requests

The concern. The policy remembers where each card is, but the library does not store anything between calls. The service must save each card's state and restore it on the next request. Three problems come with that:

  1. A Module holds the current card's state while it decides, so it is not thread-safe.
  2. Two concurrent events for the same card must not both read intake and both write their own next state.
  3. A state saved under an old policy version must not be silently restored into a new one: editing a policy can renumber or rename its states.

The code. StateStore keeps one row per card:

CREATE TABLE entity_state (
    entity_id       TEXT PRIMARY KEY,
    policy_name     TEXT NOT NULL,
    policy_version  TEXT NOT NULL,
    state           TEXT NOT NULL,
    version         INTEGER NOT NULL,   -- optimistic concurrency counter
    updated_at      TEXT NOT NULL
);

Problem 1: DecisionEngine holds one module and one lock, taken only around restore + decide:

def decide(self, saved_state, event, evidence):
    with self._lock:
        if saved_state is None:
            self.module.reset()
        else:
            self.module.restore(saved_state)
        return self.module.decide(event, evidence)

Evidence reads, database writes and network calls all happen outside the lock, and the locked work takes microseconds, so the lock is not the bottleneck. If profiling ever shows contention, use a pool of modules (a queue.Queue of modules built at start-up) or one module per single-threaded worker process.

Problem 2 is handled twice. Within a process, EntityLocks holds a lock per card id (256 shared locks chosen by a hash, so memory stays bounded). Across processes, the version column does it: the write is UPDATE ... WHERE entity_id = ? AND version = ?, and if another process got there first it changes zero rows, the transaction rolls back, and the decision is retried with the fresh state, up to three times, then HTTP 409. On Postgres, SELECT ... FOR UPDATE works too.

Problem 3: if the stored policy_version differs from the running one, the service refuses with HTTP 409 policy_version_mismatch. Migrate saved states (map old state names to new ones) before restoring across a policy change. State and policy versions explains why and how.

State is saved even when the verdict is abstain. The policy advances on every event it recognises; only the side effect is withheld.

Try it. Send a second suspicious event for the ring card that was just blocked:

curl -s localhost:8080/decide -d '{"entity_id": "acct_000",
  "event": {"name": "suspicious", "payload": {"amount_usd": 95}},
  "idempotency_key": "auth-0004", "candidate_model_score": 0.1}' \
  | jq '{routing, verdict: .record.verdict, proposed_action: .record.proposed_action,
         reason: .record.reason, state_before, state_after}'

{
  "routing": "await_next_event",
  "verdict": "abstain",
  "proposed_action": null,
  "reason": "event 'suspicious' is not handled in state 'review'; nothing proposed",
  "state_before": "review",
  "state_after": "review"
}

The card is already in review, and the policy has no suspicious step out of review, so it proposes nothing and the service does nothing: a card is not blocked twice. If your product needs a fresh decision per authorisation, make entity_id the authorisation rather than the card, or add a step back to intake in the policy (step 13).

Step 6: the audit log

The concern. Every decision must be explainable later, by card, by policy version and by which check failed, and the log must not be editable after the fact.

The code. AuditLog.append writes one row per decision to the decisions table: request id, card id, idempotency key, policy name and version, settings version (parameters_fingerprint), event name, state before and after, verdict, action, proposed action, routing, whether it was enforced, the names of the failed checks, and the full record.to_dict() and evidence.to_dict() as JSON. The JSON columns load back with DecisionRecord.from_dict and EvidenceSnapshot.from_dict, so a replay job can rebuild any decision and run it against a candidate policy. Two triggers make the table append-only:

CREATE TRIGGER decisions_are_append_only_update
    BEFORE UPDATE ON decisions
    BEGIN SELECT RAISE(ABORT, 'decisions is append-only'); END;

On Postgres, revoke UPDATE and DELETE on the table from the service's role.

Try it.

curl -s localhost:8080/decisions/acct_000 \
  | jq '{current_state, decisions: [.decisions[] | {decision_id, idempotency_key,
         event_name, state_before, state_after, verdict, routing}]}'

{
  "current_state": "review",
  "decisions": [
    {
      "decision_id": 1,
      "idempotency_key": "auth-0001",
      "event_name": "suspicious",
      "state_before": null,
      "state_after": "review",
      "verdict": "act",
      "routing": "execute"
    },
    {
      "decision_id": 4,
      "idempotency_key": "auth-0004",
      "event_name": "suspicious",
      "state_before": "review",
      "state_after": "review",
      "verdict": "abstain",
      "routing": "await_next_event"
    }
  ]
}

Each entry also has request_id, policy_version, parameters_fingerprint, failed_channels, created_at and the full record. ?limit=N returns the most recent N.

Step 7: idempotency and exactly-once side effects

The concern. Clients retry: load balancers time out and resend, Kafka redelivers, mobile apps retry on a flaky network. Without protection, a retry advances the policy again, writes a second audit row, and blocks the card twice. Separately, calling the card processor from inside the request can leave the world and the database disagreeing: the call succeeds and the commit fails, or the other way round.

The code. Two tables, both written in the decision's single transaction (Database.transaction is BEGIN IMMEDIATE ... COMMIT, rolled back on any exception):

  • idempotency_keys stores the caller's key, a hash of the request, and the exact response. A repeated key with the same request returns the stored response byte for byte, with the header Idempotent-Replay: true, and nothing else happens. A repeated key with a different request is a client bug and gets HTTP 409. If two processes race on the same key, the loser's insert fails, its transaction rolls back, and it returns the winner's stored response.
  • outbox stores the side effect to carry out (block for acct_000, with effect_key = "decision:1"). The service never calls the processor itself. A separate relay (a loop, a cron job, a change-data stream) reads pending rows, calls the processor with effect_key as its idempotency key, and marks them sent. A crashed relay re-sends; the processor ignores the duplicate key.

The rule that ties it together: the state update, audit row, review or outbox row, and idempotency key commit together or not at all.

Try it. Repeat the very first request exactly:

curl -s -i localhost:8080/decide -d '{"entity_id": "acct_000",
  "event": {"name": "suspicious", "payload": {"amount_usd": 840, "merchant": "electronics"}},
  "idempotency_key": "auth-0001", "candidate_model_score": 0.1}' | sed -n 1,8p

HTTP/1.0 200 OK
Server: GroundedDecisionReference/1.0 Python/3.12.13
Date: Fri, 09 Oct 2026 16:56:27 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 5356
X-Request-Id: 57146e14-3836-401e-8392-6693a619db4d
Idempotent-Replay: true

Reuse the key for a different request:

curl -s -w '\nHTTP %{http_code}\n' localhost:8080/decide -d '{"entity_id": "acct_000",
  "event": {"name": "clean"}, "idempotency_key": "auth-0001", "candidate_model_score": 0.1}'

{
  "error": "idempotency_key_reused",
  "message": "Idempotency key 'auth-0001' was already used for a different request. Use a new key for a new request."
}

HTTP 409

The outbox still holds exactly one block:

curl -s localhost:8080/outbox

{
  "outbox": [
    {
      "action": "block",
      "authority": "policy",
      "created_at": "2026-10-09T16:56:27.933889+00:00",
      "decision_id": 1,
      "effect_key": "decision:1",
      "entity_id": "acct_000",
      "outbox_id": 1,
      "review_id": null,
      "status": "pending"
    }
  ]
}

Use a key that is naturally unique per request (the authorisation id is ideal), and delete keys older than your longest retry horizon on a schedule.

Step 8: the review queue, and closing the loop

The concern. An abstain with a proposal is a case for a person. The reviewer needs to see why the service held back, and their answer should improve the next decision.

The code. ReviewQueue.enqueue stores the proposal, the reason, the names of the failed checks and the health numbers. A reviewer who sees "neighbour agreement failed, three links" decides differently from one who sees "unusual case: score unlike anything confirmed recently". Resolving a review (POST /reviews/<id>/resolve with {"outcome": "confirmed_fraud" | "cleared", "reviewer": "..."}) does four things in one transaction:

  1. marks the review resolved, with the outcome and reviewer;
  2. writes the outcome into review_outcomes, the table the evidence provider reads, so this person's judgement becomes evidence for the next decision about any linked account;
  3. records the held decision's outcome in decision_outcomes: the proposed block was right if the reviewer confirmed fraud and wrong if they cleared the account. This is how held cases get labels for learning (step 11);
  4. writes the action the reviewer authorised (block or pass) to the outbox with authority = "human", and resets the card's state to intake, because the case is closed.

Try it. List the open reviews; there are two, acct_777 (review 1) and acct_900 (review 2). Here is the second:

curl -s localhost:8080/reviews | jq '.reviews[] | select(.review_id == 2)'

{
  "created_at": "2026-10-09T16:56:27.957501+00:00",
  "decision_id": 3,
  "entity_id": "acct_900",
  "failed_channels": [
    "reconstruction",
    "threshold"
  ],
  "health": {
    "conformal_p": 1.0,
    "d_lambda_sq": 0.1666666666666666,
    "game_solved": true,
    "in_operational_window": true,
    "in_theorem_window": false,
    "mixer_status": "idle",
    "p_hat": 0.5,
    "p_threshold": 0.08,
    "reconstruction_margin": 0.05,
    "star_height": 0,
    "support": null,
    "theorem_window_vacuous": true,
    "threshold_width_operational": 1.6749170738973782,
    "threshold_width_theorem": 439069.4614197543
  },
  "outcome": null,
  "proposed_action": "block",
  "reason": "d λ² = 0.1667 vs 1+0.050 (at or below KS; abstain); evidence density inside operational window (theorem width=4.39e+05, vacuous=True)",
  "resolved_at": null,
  "review_id": 2,
  "reviewer": null,
  "status": "open"
}

failed_channels uses the record's names for the checks: reconstruction is neighbour agreement and threshold is the knife-edge check. threshold_width_operational is 1.67, wider than the whole 0-to-1 range, which is why the 4-account chain could not pass.

Resolve it:

curl -s localhost:8080/reviews/2/resolve \
  -d '{"outcome": "confirmed_fraud", "reviewer": "alice@example.com"}' \
  | jq '{action, outbox_id, review: (.review | {review_id, status, outcome, reviewer}), decision_outcome}'

{
  "action": "block",
  "outbox_id": 2,
  "review": {
    "review_id": 2,
    "status": "resolved",
    "outcome": "confirmed_fraud",
    "reviewer": "alice@example.com"
  },
  "decision_outcome": {
    "action_was_correct": true,
    "decision_id": 3,
    "recorded": true
  }
}

curl -s localhost:8080/decisions/acct_900 now reports "current_state": "intake". Resolving the same review again returns HTTP 409 already_resolved. GET /reviews?status=resolved and ?status=all list closed cases. In production, the review row is what you send to your case tool, and the resolve endpoint is the webhook that tool calls back.

Step 9: observability

The concern. The act rate will move, and you need to know why before you page anyone. A broken evidence job (empty graphs, so neighbour agreement fails everywhere) and data sitting on the decision line (the knife-edge check fails) both look like "act rate dropped" if you only watch the verdict.

The code. Each decision writes one JSON log line with the ids you will search by, the verdict and routing, the failed checks, and the health numbers that explain most abstains:

2026-10-09 12:56:27,958 {"conformal_p": 1.0, "d_lambda_sq": 0.1667, "enforced": true, "entity_id": "acct_900", "event": "suspicious", "failed_channels": ["reconstruction", "threshold"], "idempotency_key": "auth-0003", "latency_ms": 2.071, "message": "decision", "p_hat": 0.5, "p_threshold": 0.08, "parameters_fingerprint": "settings@13447e40df90", "policy_version": "fraud_review@557323c7e95c", "request_id": "9bca399a-1dd9-41c9-ac59-f57f09cd91ef", "routing": "human_review", "state_after": "review", "verdict": "abstain"}

The event payload is deliberately not logged: it can hold personal data, and the audit table already has it under access control. Callers can send X-Request-Id to correlate with their own logs; otherwise the service generates one and returns it in the response header.

ServiceMetrics keeps counters in the Prometheus text format. After the steps above:

curl -s localhost:8080/metrics

# TYPE grounded_decision_decisions_total counter
grounded_decision_decisions_total{verdict="abstain",routing="await_next_event"} 1
grounded_decision_decisions_total{verdict="abstain",routing="human_review"} 2
grounded_decision_decisions_total{verdict="act",routing="execute"} 1
# TYPE grounded_decision_channel_failures_total counter
grounded_decision_channel_failures_total{channel="reconstruction"} 2
grounded_decision_channel_failures_total{channel="threshold"} 2
# TYPE grounded_decision_decision_latency_seconds summary
grounded_decision_decision_latency_seconds_sum 0.008547
grounded_decision_decision_latency_seconds_count 4
# TYPE grounded_decision_idempotent_replays_total counter
grounded_decision_idempotent_replays_total 1
# TYPE grounded_decision_state_conflicts_total counter
grounded_decision_state_conflicts_total 0
# TYPE grounded_decision_outcomes_recorded_total counter
grounded_decision_outcomes_recorded_total{action="correct"} 1
grounded_decision_outcomes_recorded_total{action="wrong"} 0
# TYPE grounded_decision_enforcing gauge
grounded_decision_enforcing 1

outcomes_recorded_total counts the review resolution's outcome; enforcing is 0 in shadow mode. With learned parameters loaded the risk monitor adds its gauges (step 11).

Alert on failures per check, not on the act rate alone: a jump in channel_failures_total{channel="reconstruction"} with flat traffic usually means the evidence source returned empty or stale graphs. GET /health doubles as the liveness probe (it queries the database) and reports the policy version, so a deploy dashboard can confirm every instance runs the same one. Operate and monitor lists what to watch and what each signal means, and Troubleshoot: why did it abstain? goes from a symptom to the check and the fix.

Step 10: roll out in shadow mode first

The concern. You do not switch a fraud system to a new decision engine on day one. You run it beside the existing rule, compare, and only then let it act.

The code. --shadow turns enforcement off. Every decision is still made, stored, audited and counted, and state still advances, so the shadow policy tracks each card exactly as it would in production. But no review is opened and nothing is written to the outbox: the existing system stays in charge. GET /report compares, for the current policy version, the grounded service with the naive rule "carry out whatever the policy proposes" (which is what a score cut-off does); withheld_by_grounding counts the decisions the service would have held.

Try it. Stop the first server (Ctrl-C; it drains in-flight requests, prints reference service: interrupted (Ctrl-C) and exits with status 130) or use another port:

uv run python -m grounded_decision.examples.reference_service \
  --seed-demo-data --shadow --port 8090 --database /tmp/shadow.sqlite3
# in another terminal:
for acct in acct_000 acct_005 acct_900 acct_902; do
  curl -s localhost:8090/decide -d "{\"entity_id\": \"$acct\",
    \"event\": {\"name\": \"suspicious\"}, \"idempotency_key\": \"shadow-$acct\",
    \"candidate_model_score\": 0.1}" | jq -c '{entity_id, routing, enforced, outbox_id, review_id}'
done

{"entity_id":"acct_000","routing":"execute","enforced":false,"outbox_id":null,"review_id":null}
{"entity_id":"acct_005","routing":"execute","enforced":false,"outbox_id":null,"review_id":null}
{"entity_id":"acct_900","routing":"human_review","enforced":false,"outbox_id":null,"review_id":null}
{"entity_id":"acct_902","routing":"human_review","enforced":false,"outbox_id":null,"review_id":null}

curl -s localhost:8090/report

{
  "by_routing": {
    "execute": 2,
    "human_review": 2
  },
  "by_verdict": {
    "abstain": 2,
    "act": 2
  },
  "decisions": 4,
  "failed_channels": {
    "reconstruction": 2,
    "threshold": 2
  },
  "grounded_acts": 2,
  "naive_acts": 4,
  "policy_version": "fraud_review@557323c7e95c",
  "withheld_by_grounding": 2
}

The naive rule would have blocked all four cards; the service blocks the two in the ring and holds the two in the chain. The report is per policy version, so each threshold or policy change starts a fresh comparison. Roll out safely gives the full plan: how long to shadow, how to compare against real outcomes, and the order in which to switch enforcement on.

Step 11: learn the settings from your own outcomes

The concern. Every number the checks use so far was typed in: the evidence-strength margin (0.05), the knife-edge window, the unusualness level (0.10) and the 40 calibration scores. Nobody can say what error rate they buy. Worse, the calibration scores are risk-model scores of confirmed-fraud cases, while each request's candidate_model_score is the score of this case for whatever action is proposed: a clean authorisation is compared with fraud cases. What you can say is how often an automatic decision may be wrong ("at most 10%"). This step lets the service learn everything else from its own decisions and their outcomes, with a certificate, and then watch that the certificate still holds.

The loop: decide, record each decision's outcome, export the history, learn, restart with the learned parameters, monitor.

POST /decide ──> decisions ──> POST /outcomes (and review resolutions)
                                     │
GET /history.jsonl  <────────────────┘
      │
make learn HISTORY=history.jsonl MAX_RISK=0.1   ──> parameters.json
      │
restart with --parameters parameters.json ──> GET /parameters, GET /health

The code.

  • POST /outcomes with {"decision_id", "action_was_correct"} records what eventually turned out to be true about the proposed action of any decision, carried out or held: a chargeback on a passed card, a customer complaint about a block, a fraud review. It is idempotent per decision (the same outcome again returns "recorded": false; the opposite outcome is HTTP 409, because an outcome is never overwritten). Resolving a review records the held decision's outcome automatically (step 8).
  • GET /history.jsonl exports every decision that proposed an action and has an outcome, one line per decision, in exactly the format grounded-decision learn reads: the evidence the decision was made on, the proposed action, action_was_correct, case_id = "decision:<id>", and was_acted_on (true when the decision was enforced and routed to execute or execute_degraded). The learner uses was_acted_on to warn when the history has outcomes only for decisions that were carried out.
  • --parameters parameters.json loads the learned parameters at start-up. The module is built with the learned check settings, and for each request the evidence gets the calibration scores of past cases where the same proposed action turned out correct: a block is compared with blocks that were right, a pass with passes that were right. The service refuses to start with parameters that are not certified, unless you pass --allow-uncertified (then every decision is held, which is what uncertified parameters do).
  • GET /parameters shows the parameters in force, their full provenance and both fingerprints. The learned fingerprint (params@…) is what grounded-decision learn printed; the settings fingerprint (settings@…) is what every decision carries.
  • RiskGuard feeds every outcome of a decision made under the loaded settings to a RiskMonitor built for the learned target. GET /health and GET /metrics show its status, its bounds on the recent error rate and how many outcomes it has seen.

Try it. This needs more traffic than the ring and the chain, so the service ships a synthetic workload: --seed-learning-demo loads 2,000 separate device neighbourhoods (from the parameter_learning example), each around one flagged account, with the truth known. Most accounts in these neighbourhoods are clean and their links are partly random, so measure agreement against the neighbourhood's own label mix (--agreement-baseline label_mix): with the default measure, random links between mostly-clean accounts look like agreement. The parameter learning case shows the effect. Stop the first server, then:

uv run python -m grounded_decision.examples.reference_service \
  --seed-learning-demo --agreement-baseline label_mix \
  --port 8081 --database /tmp/learning.sqlite3

In a second terminal, send the first 1,600 authorisations and, for each one, the outcome your chargeback and review systems would report later. This is the only part that stands in for your systems:

uv run python - 0 1600 <<'PY'
import json, sys, urllib.request
from grounded_decision.examples.parameter_learning import synthetic_authorisations
from grounded_decision.examples.reference_service import LEARNING_DEMO_CASES, LEARNING_DEMO_SEED

def post(path, body):
    request = urllib.request.Request("http://127.0.0.1:8081" + path, json.dumps(body).encode())
    with urllib.request.urlopen(request) as response:
        return json.load(response)

start, stop = int(sys.argv[1]), int(sys.argv[2])
cases = synthetic_authorisations(LEARNING_DEMO_CASES, seed=LEARNING_DEMO_SEED)[start:stop]
verdicts = {}
for case in cases:
    decision = post("/decide", {"entity_id": case.flagged_account,
                                "event": {"name": case.event_name},
                                "idempotency_key": f"auth-{case.case_id}",
                                "candidate_model_score": case.unusualness_score})
    # Later, when the truth is known (chargeback, review, customer contact):
    proposed = decision["record"]["proposed_action"]
    post("/outcomes", {"decision_id": decision["decision_id"],
                       "action_was_correct": case.action_was_correct(proposed)})
    verdicts[decision["record"]["verdict"]] = verdicts.get(decision["record"]["verdict"], 0) + 1
print(f"{len(cases)} decisions, outcomes recorded for all:", verdicts)
PY

1600 decisions, outcomes recorded for all: {'abstain': 1259, 'act': 341}

With the hand-set settings the service acted on 341 of the 1,600, and 25 of those 341 (7.3%) turned out wrong. Export the history and look at one line:

curl -s localhost:8081/history.jsonl > history.jsonl
head -1 history.jsonl | jq -c '{case_id, proposed_action, action_was_correct, was_acted_on,
  evidence: (.evidence | {n, degree, lambda_hat, p_hat, label_score})}'

{"case_id":"decision:1","proposed_action":"pass","action_was_correct":true,"was_acted_on":false,"evidence":{"n":11,"degree":6.545454545454546,"lambda_hat":-0.058823529411764705,"p_hat":0.6545454545454545,"label_score":0.371}}

This decision was held (was_acted_on: false), and its outcome is in the history anyway: the script recorded outcomes for every decision. That is what lets the learner judge the held cases too.

Learn. MAX_RISK is the business input: the largest share of automatic decisions you accept being wrong. First ask for 2%:

make learn HISTORY=history.jsonl MAX_RISK=0.02

Goal: at most 2.0% wrong of the decisions it acts on, certified with 95% confidence.
History: 1600 labelled cases, split calibration 480, risk 800, holdout 320 (seed 0, dataset data@f3b0ff0e9b59).
Tried 1200 combinations of settings; 0 were certified.
Procedure: fixed-sequence testing. The combinations were ordered on the calibration part (most clearly safe there first), then tested in that order on the risk part, each at significance 0.05, stopping at the first failure.
No combination could be certified, so these settings hold every decision. Either there is too little history (even with zero errors, at least 149 acted-on cases are needed in the risk part), or every setting that acts often enough makes too many mistakes. Fixes: more labelled history, a better definition of related cases or a better model score, or a higher acceptable risk.
Warning: the first combination tested (the most clearly safe one on the calibration part) failed on the risk part, so testing stopped: the target is too strict for this history, or there is too little history to show it is met
Fingerprint: params@ff131423f2e2
Wrote parameters.json
make: *** [learn] Error 2

1,600 cases cannot certify 2% here, and the command exits with status 2. It still writes parameters.json, but those parameters hold every decision, and the service refuses to start with them:

python -m grounded_decision.examples.reference_service: error: the parameters have status 'no_safe_configuration', not 'certified': no setting could be shown to meet the target, so they hold every decision. Collect more labelled history, or pass --allow-uncertified to run anyway (everything will be held).

Ask for 10%:

make learn HISTORY=history.jsonl MAX_RISK=0.1

Goal: at most 10.0% wrong of the decisions it acts on, certified with 95% confidence.
History: 1600 labelled cases, split calibration 480, risk 800, holdout 320 (seed 0, dataset data@f3b0ff0e9b59).
Tried 1200 combinations of settings; 540 were certified.
Procedure: fixed-sequence testing. The combinations were ordered on the calibration part (most clearly safe there first), then tested in that order on the risk part, each at significance 0.05, stopping at the first failure.
Chosen: reconstruction margin -1 (evidence strength must exceed 0), window prefactor 0 (knife-edge check off), decision line each case's own, conformal alpha 0.2, minimum support 0 (support check off).
Certificate: with 95% confidence the true error rate is at most 10.0%, provided future cases are independent draws from the same process as this history.
Risk part (the data the settings were chosen on, so optimistic): acted on 495 of 800, 33 wrong.
Hold-out (never used for choosing): acted on 194 of 320 (60.6%), 16 wrong; error rate at most 12.26%.
Fingerprint: params@c1b67d6a0eda
Wrote parameters.json

The learner switched the evidence-strength check off (margin -1: strength must only exceed 0) and the knife-edge check too (on this data they held cases without preventing errors at a 10% target), kept the unusual-case check at alpha 0.2, and replaced the 40 hand-set calibration scores with the scores of past cases where each action was right. The support check stays off: the service does not measure support, so there is nothing to learn. The hold-out line is an honest check on 320 cases the learner never saw; with 194 acts there, its upper bound (12.26%) is wider than the target, which is expected for so few cases. Parameter learning explains every line, and Learning settings with a guarantee explains the "Procedure" line.

Restart the service with the learned parameters (Ctrl-C the first, keep the same database):

uv run python -m grounded_decision.examples.reference_service \
  --agreement-baseline label_mix --port 8081 --database /tmp/learning.sqlite3 \
  --parameters parameters.json

2026-10-09 12:59:59,102 serving http://127.0.0.1:8081 policy=fraud_review@557323c7e95c settings=settings@1c275e611777 learned=params@c1b67d6a0eda mode=enforce database=/tmp/learning.sqlite3

The policy version is unchanged, so every saved card state is still valid; only the settings version moved. Check what is in force:

curl -s localhost:8081/parameters | jq '{parameters_fingerprint, learned_fingerprint,
  check_settings, calibration_source, status: .learned.status,
  calibration_scores: (.learned.parameters.calibration_scores | map_values(length)),
  provenance: (.learned.provenance | {target, procedure, seed, dataset_fingerprint, split_sizes})}'

{
  "parameters_fingerprint": "settings@1c275e611777",
  "learned_fingerprint": "params@c1b67d6a0eda",
  "check_settings": {
    "conformal_alpha": 0.2,
    "min_support": 0.0,
    "operational_fk_prefactor": 0.0,
    "reconstruction_margin": -1.0
  },
  "calibration_source": "learned: learned.parameters.calibration_scores, per proposed action",
  "status": "certified",
  "calibration_scores": {
    "block": 204,
    "pass": 175
  },
  "provenance": {
    "target": {
      "confidence": 0.95,
      "max_risk": 0.1,
      "measure": "error_rate_among_acts"
    },
    "procedure": "fixed_sequence",
    "seed": 0,
    "dataset_fingerprint": "data@f3b0ff0e9b59",
    "split_sizes": {
      "calibration": 480,
      "holdout": 320,
      "risk": 800
    }
  }
}

Send the remaining 400 authorisations with the same script (uv run python - 1600 2000 <<'PY' … PY):

400 decisions, outcomes recorded for all: {'act': 266, 'abstain': 134}

The service now acts on 66.5% of authorisations (266 of 400) where the hand-set settings acted on 21.3%, and 13 of the 266 (4.9%) were wrong, against 7.3% before. Watch the monitor:

curl -s localhost:8081/health | jq '{enforcement, enforcement_reason, learned_parameters, risk_monitor}'

{
  "enforcement": "enforce",
  "enforcement_reason": "enforcing",
  "learned_parameters": {
    "fingerprint": "params@c1b67d6a0eda",
    "status": "certified"
  },
  "risk_monitor": {
    "errors": 13,
    "lower_bound": 0.029147,
    "status": "within_target",
    "target": {
      "confidence": 0.95,
      "max_risk": 0.1,
      "measure": "error_rate_among_acts"
    },
    "trials": 266,
    "tripped": false,
    "upper_bound": 0.076579,
    "window": 500
  }
}

The recent error rate among acted-on decisions is between 2.9% and 7.7% (95% bounds), so the monitor says it is below 10% (within_target); with fewer outcomes it would say uncertain. The same numbers are on GET /metrics as grounded_decision_risk_monitor_* gauges.

The fail-safe: a high error rate switches the service to shadow

When the monitor's lower bound on the recent error rate rises above the target ("status": "above_target"), the world no longer looks like the history the parameters were certified on. The service then switches itself to shadow mode: it keeps deciding and recording everything, but writes nothing to the outbox and opens no reviews. GET /health says "enforcement": "shadow" with the reason, and grounded_decision_enforcing drops to 0: alert on it. The switch stays until you restart with re-learned parameters (restarting with the same ones rebuilds the monitor from the database and trips again). tests/test_reference_service.py shows it happening.

What the certificate does and does not cover:

  • Record outcomes for held decisions too. If only acted-on decisions get outcomes, the history describes only the cases the current settings already liked, and the learner cannot tell whether holding the others was necessary (the selective labels problem). Review resolutions label held cases for you; label the rest when the truth arrives. The exported was_acted_on lets the learner notice: when every case with a known outcome was acted on, learn prints Warning: every case with a known outcome was acted on; outcomes of held decisions are missing, so the certificate describes a filtered population.
  • Independent cases. The certificate assumes the cases in the history are independent. Decisions about accounts in the same fraud ring share their evidence and usually their outcome. Keep one decision per cluster of related cases when you export (the demo's neighbourhoods are separate by construction).
  • Re-learn, don't patch. The calibration scores and the check settings were certified together. Refreshing only the calibration scores, say daily, changes the configuration the certificate was about. Re-run make learn on the fuller history and restart instead; the settings version changes, saved card states do not.

Step 12: concurrency and scaling

The reference server runs one thread per request, which is why the engine lock, the per-card locks and one short-lived database connection per unit of work exist. To scale out:

  • More workers. Put the same DecisionService behind a production server with several processes (FastAPI under uvicorn or gunicorn; Integrate into a service shows that wrapper and a Kafka consumer). Each process builds its own engine at start-up. Same-card races across processes are caught by the state version column, and idempotency-key races by the primary key.
  • SQLite to Postgres. SQLite in write-ahead-log mode handles one writer at a time: fine for one host and a few hundred decisions per second. Beyond that, replace Database with a connection pool. The SQL is portable apart from four details listed in the Database docstring: ? placeholders become %s, AUTOINCREMENT becomes an identity column, BEGIN IMMEDIATE becomes BEGIN, and the append-only triggers become revoked privileges.
  • Hot cards. With many processes, the in-process locks only reduce retries; correctness comes from the version check. If one card is very busy, partition the event stream by card id (a Kafka partition key) so one consumer owns each card.
  • Shutdown. Ctrl-C and SIGTERM both stop the server cleanly: open transactions roll back, in-flight requests drain, and the last line on stderr says which signal stopped it (reference service: interrupted (Ctrl-C), exit status 130, or reference service: stopped (SIGTERM), exit status 0). There is no traceback at any stage, including during start-up and seeding.
  • Evidence. At scale the evidence walk is the expensive part, not decide. Precompute snapshots in a batch job and cache them per card.

Step 13: adapt it to your domain

Three things change. Everything else (state, audit, idempotency, outbox, review queue, metrics, shadow mode, HTTP) is generic and stays.

  1. The policy. Replace build_fraud_engine with a function that builds your Module, from policy text with Module.from_expr or with the Statechart builder (Write a policy). Decide which events move a case forward, which state ends it, and whether a resolved case returns to the start. The fingerprint and the list of valid events follow automatically.
  2. The evidence provider. Implement EvidenceProvider against your data: what the related cases are and what links them (accounts sharing devices, sensors agreeing about a machine, reviewers agreeing about a post), where the outcomes come from, and what the calibration scores are (or learn them, step 11). Return your settings from evidence_parameters. Build evidence from your data and Choose thresholds and calibration cover these choices; the Example library maps many domains.
  3. What each routing does. route_decision rarely changes, but what follows does: what the outbox relay calls for an execute (freeze a card, page an engineer, approve a loan), where a human_review row goes (your case tool), and what a resolved review means (REVIEW_OUTCOME_LABELS and ACTION_FOR_REVIEW_OUTCOME are the fraud-specific mappings). Rename candidate_model_score if your domain calls it something else.

Then replace the demo seed with your own fixtures, copy tests/test_reference_service.py, and change its expectations to your cases: one that should act, one that should be held for review, one replay and one bad request.

What this service does not do

  • It does not measure support. The evidence provider does not tell the library which outcome each action implies, so the support check passes with support not measured on every decision, and a clean event on a ring account would act pass. To turn it on, pass supporting_label=fraud_review.SUPPORTING_LABEL[proposed_action] to device_graph_evidence in evidence_for (use module.peek_proposed_action to find the proposed action first); see Do the related cases support this action?
  • The unusual-case check is not exercised in steps 0 to 10. Every request there sends 0.1, which is below every calibration score. Send a score above 1.18 to see it hold.
  • Small groups of related accounts are held almost always. That is the knife-edge check working as designed; plan reviewer capacity for it.

Next