Skip to content

Roll out safely

Nobody should switch a fraud, triage or moderation system to a new decision engine on day one and trust it. The safe path is to run the grounded service beside whatever decides today, compare what each would have done against the outcomes you later learn, and only then let it act, one step at a time. This page gives that plan, with the commands for the reference service, and covers the part teams most often get wrong: changing the policy or a threshold once entities have saved state.

Who this is for: the engineer running the launch and the domain owner approving each step.

What you need:

The plan at a glance

flowchart LR
    A["1. Shadow<br/>decide and record,<br/>execute nothing"] --> B["2. Compare<br/>against outcomes<br/>and the current rule"]
    B --> C["3. Enforce holds<br/>held cases go to people;<br/>current rule still acts"]
    C --> D["4. Enforce acts<br/>for a slice of entities"]
    D --> E["5. Full enforcement"]
    B -.->|numbers outside the agreed range| A
    D -.->|wrong acts or queue overload| C

Each arrow is a decision with a named owner and a number to check. Write the numbers down before you start.

Step 1: run in shadow mode

Shadow mode decides, stores, audits and counts everything, and advances each entity's state exactly as it would in production, but opens no review and writes nothing to the outbox. The existing system stays in charge.

make service SERVICE_ARGS="--seed-demo-data --shadow"

Call it from the existing path without waiting for the answer (fire and forget, or from a copy of the event stream), with the same idempotency key the existing system uses, so retries line up. GET /health reports "enforcement": "shadow". Step 10 of Build a decision service shows a shadow run over curl.

How long: at least one full outcome cycle (if chargebacks take 30 days to arrive, 30 days of decisions plus the wait), and long enough that the live act rate's confidence range is narrower than the band you agreed. Weekly patterns need at least two weeks.

Step 2: compare against outcomes and the current rule

GET /report counts, for the running policy version, what a naive rule ("do whatever the policy proposes") would have done and how often the grounded service withheld an action. Then join the audit table to the outcomes you learned. This script does both against the demo data, in process (save as shadow.py, run uv run python shadow.py):

import json
import logging
import tempfile
from pathlib import Path

from grounded_decision.examples.reference_service import (
    build_service,
    parse_decide_request,
)

logging.disable(logging.INFO)  # keep the per-decision log lines out of this printout
database = Path(tempfile.mkdtemp()) / "shadow.sqlite3"
service = build_service(database, seed_demo=True, enforce=False)  # --shadow

accounts = [f"acct_{i:03d}" for i in range(16)] + [f"acct_{i}" for i in range(900, 904)]
for account in accounts:
    body = {
        "entity_id": account,
        "event": {"name": "suspicious"},
        "idempotency_key": f"shadow-{account}",
        "candidate_model_score": 0.1,
    }
    service.decide(
        parse_decide_request(body, service.engine.valid_event_names), account
    )

print(json.dumps(service.report(), sort_keys=True))

# Join what the service would have done with what the review later found.
with service.database.connection() as connection:
    rows = connection.execute(
        """
        SELECT d.routing, COALESCE(o.outcome, 0) AS fraud, COUNT(*) AS cases
        FROM decisions AS d
        LEFT JOIN review_outcomes AS o ON o.account_id = d.entity_id
        WHERE d.policy_version = ? AND d.enforced = 0
        GROUP BY d.routing, fraud ORDER BY d.routing, fraud""",
        (service.engine.policy_version,),
    ).fetchall()
for row in rows:
    print(
        f"{row['routing']:13s} later found {'fraud' if row['fraud'] else 'clean'}: {row['cases']}"
    )

{"by_routing": {"execute": 16, "human_review": 4}, "by_verdict": {"abstain": 4, "act": 16}, "decisions": 20, "failed_channels": {"reconstruction": 4, "threshold": 4}, "grounded_acts": 16, "naive_acts": 20, "policy_version": "fraud_review@557323c7e95c", "withheld_by_grounding": 4}
execute       later found clean: 3
execute       later found fraud: 13
human_review  later found clean: 2
human_review  later found fraud: 2

(In this demo the "later" outcomes are the seeded review table the evidence itself is built from; in production use outcomes learned after each decision.)

Read it as three questions:

  1. Would the service act wrongly? It would block all 16 ring accounts, including the 3 later found clean: those accounts sit in a dense ring of fraud, and the checks measure how informative the ring is, not whether each account is guilty. The naive rule would also wrongly block the 2 clean accounts in the thin chain; the grounded service sends those to a person. Decide with the domain owner whether 3 wrong blocks out of 16 acts is acceptable, and if not, change the policy or the evidence, not the threshold of convenience.
  2. Are holds held for the reasons you expect? failed_channels should match what Estimate impact predicted. A check failing far more often than predicted usually means a broken evidence source (Troubleshoot).
  3. Is the volume what you planned? Compare the live act and hold rates with the range from the replay. Outside it, go back to shadow and find out why before going further.

The report is per policy version, so every policy change starts a fresh comparison. A threshold or learned-parameter change does not; compare the settings version (parameters_fingerprint, on every decision) when you need to split the numbers by it.

Learn the settings from shadow-mode outcomes

Shadow mode is also how you collect the labelled history the learner needs, from your own traffic, before the service has acted on anything. Every shadow decision is in the audit table with its evidence (evidence_json) and its proposed action. Once outcomes arrive, join them on, write one JSON line per decision that proposed an action ({"evidence": ..., "proposed_action": ..., "action_was_correct": ...}), and learn:

uv run grounded-decision learn history.jsonl --max-risk 0.05 --output params.json

Learn parameters from your history walks through the output and loading the result. Three points specific to rollout:

  • Shadow mode avoids most selective labels. The service executes nothing, so its holds do not decide which cases get an outcome. The existing system still does: if it only reviews what it flags, you learn outcomes for those cases only. Sample the rest for review.
  • Learn before step 3, and again before step 5. The first run sets the values you enforce; the second, on outcomes from the enforced slice, confirms them. Each new set of parameters is a setting change (below).
  • Use the hold-out line for the go/no-go. "On 600 cases it had not seen, it acted on 18.7% and was wrong once" (the how-to's worked example) is the kind of number the domain owner signs, next to the review-queue size from Estimate impact. If nothing is certified, stay in shadow mode and collect more outcomes.

Step 3: enforce holds first

Turn enforcement on (restart without --shadow) but do not run the outbox relay yet, or run it in a dry-run mode that logs instead of calling the downstream system. The service now opens reviews for held cases and records acts in the outbox, while the existing rule keeps executing. The caller treats a human_review response as "do not auto-act, wait for the reviewer". This tests reviewer capacity and the review tool with real traffic, while the old rule still owns every automatic action.

Step 4: enforce acts for a slice

Let the relay execute outbox rows for a slice of entities, chosen by a stable hash of the entity id so each entity is always on the same side:

import hashlib


def in_rollout(entity_id: str, percent: int) -> bool:
    bucket = int(hashlib.sha256(entity_id.encode()).hexdigest(), 16) % 100
    return bucket < percent

The relay executes a row only when in_rollout(row["entity_id"], 5); the existing rule keeps acting for the rest. Raise the slice (5%, 25%, 50%) when the wrong-act count and queue load over the last outcome cycle stay inside the agreed band. If they do not, drop back to step 3; nothing else changes.

Step 5: full enforcement

Execute every outbox row and retire the old rule's automatic path. Keep /report, the metrics and the alerts from Operate running: they are now your regression check for every future change.

Change the policy or a threshold in production

A saved state is a position in one compiled policy. The reference service versions two things separately: the policy version (policy_fingerprint: the chart's states, actions, verdicts, payoffs, priorities and arrows, and the policy text) and the settings version (parameters_fingerprint: Limits, the density cut-off, the calibration scores, the caps, learned parameters). A settings change, such as a new density cut-off or a newly learned params.json, changes only the settings version; saved states stay valid. A policy change makes every entity with saved state fail with 409 policy_version_mismatch until you migrate. The script below ships a threshold change (no 409), then a policy change (a new appeal arrow), hits the 409, migrates, and decides again:

import logging
import tempfile
from pathlib import Path

from grounded_decision import Module
from grounded_decision.examples import fraud_review
from grounded_decision.examples.reference_service import (
    ServiceError,
    build_service,
    parse_decide_request,
)

logging.disable(logging.INFO)
database = Path(tempfile.mkdtemp()) / "service.sqlite3"


def decide(service, account, event, key):
    body = {
        "entity_id": account,
        "event": {"name": event},
        "idempotency_key": key,
        "candidate_model_score": 0.1,
    }
    try:
        result = service.decide(
            parse_decide_request(body, service.engine.valid_event_names), key
        )
        return "ok"
    except ServiceError as error:
        return f"{int(error.status)} {error.error_code}"


old = build_service(database, seed_demo=True)
print("v1", old.engine.policy_version, decide(old, "acct_000", "suspicious", "k1"))

fraud_review.RING_DENSITY_THRESHOLD = 0.10  # a settings change: same policy version
same = build_service(database)
print(
    "v1'",
    same.engine.policy_version,
    same.engine.parameters_version,
    decide(same, "acct_000", "hold", "k1b"),
)

original_build = fraud_review.build


def build_with_appeal() -> Module:  # the policy change being shipped
    module = original_build()
    chart = module.chart.on("review", "appeal", "intake")  # a blocked card can appeal
    return Module(name=module.name, chart=chart, limits=module.limits)


fraud_review.build = build_with_appeal
new = build_service(database)
print("v2", new.engine.policy_version, decide(new, "acct_000", "hold", "k2"))


def migrate(database, old_version, new_version, state_map):
    """Re-stamp saved states from old_version to new_version, mapping names."""
    with database.transaction() as connection:
        rows = connection.execute(
            "SELECT entity_id, state FROM entity_state WHERE policy_version = ?",
            (old_version,),
        ).fetchall()
        for row in rows:
            connection.execute(
                "UPDATE entity_state SET state = ?, policy_version = ?, version = version + 1"
                " WHERE entity_id = ? AND policy_version = ?",
                (state_map[row["state"]], new_version, row["entity_id"], old_version),
            )
    return len(rows)


identity = {state: state for state in new.engine.module.chart.states()}
print(
    "migrated",
    migrate(
        new.database, old.engine.policy_version, new.engine.policy_version, identity
    ),
)
print("v2", decide(new, "acct_000", "hold", "k3"))

v1 fraud_review@557323c7e95c ok
v1' fraud_review@557323c7e95c settings@cc1ddd8694b5 ok
v2 fraud_review@8b135228839e 409 policy_version_mismatch
migrated 1
v2 ok

The procedure:

  1. Write the state map. For each state of the old version, the state of the new version that means the same thing. If only settings changed in a way that keeps every state's meaning (as with the new appeal arrow above), it is the identity map. If the chart changed, map by meaning; a policy written as text renumbers its states (s0, s1, …) when you edit it, so compare what each state proposes (Write a policy). An old state with no new equivalent should raise (the KeyError from state_map[...]), not be guessed.
  2. Pause writes (stop the queue consumer, or accept a few seconds of 409s that callers retry with the same idempotency key), deploy the new version everywhere, run the migration, resume.
  3. Keep the reverse map. Rolling back is the same migration in the other direction; without it, the old version 409s on every entity the new one touched.
  4. Watch the fresh report. The new version starts with an empty /report. Treat it like step 2 in miniature.

Two alternatives, each with a cost: let in-flight entities finish on the old version by routing requests by the stored version to two engines (the reference service does not do this for you), or reset every in-flight entity to the initial state (cases restart; say so to the domain owner).

Calibration refreshes and re-learned parameters change only the settings version, so they need no migration. They still change what the service accepts: ship them through shadow mode like any other change, and note that GET /report restarts its comparison only on a policy version change.

Result

A service that went live in measured steps, each approved against numbers agreed in advance, and a tested procedure for every future policy or threshold change.

What can go wrong

  • Shadow traffic that is not production traffic. A sampled tap that drops retries or a region misses exactly the cases that matter. Check the shadow decision count against the existing system's.
  • Comparing against outcomes you only have for reviewed cases. The held cases get outcomes from reviewers; acted cases may not (a blocked card is rarely re-examined). Sample some acted cases for review during steps 2 to 4, or the wrong-act count will look better than it is, and settings learned from that history are certified for the wrong population (selective labels).
  • Treating withheld_by_grounding as the benefit. It counts holds, not correct holds. Only the join with outcomes says whether holding was right.
  • Migrating without pausing. An old process still running writes old states under the old version after the migration; they 409 on the new version.

Next