Incident response: run the playbook or page a person¶

An incident bot that can roll back a deploy or shift traffic on its own saves minutes on a real outage. The same bot, firing on a monitor that is only flapping, causes an outage of its own and wakes someone up to undo it. This page shows how the library lets the bot mitigate when the surrounding monitors look like a real incident, and hand the alert to the on-call engineer when they don't. It is written for SRE leads and on-call owners; no maths is needed to follow it. The runnable example id is incident_response.

At a glance


Decision	On an alert, run the mitigation playbook automatically, or hold it for a person
Owner	The SRE team that owns the bot and its playbooks; the on-call engineer handles holds
Today's rule	Mitigate on every alert
What this adds	Looks at the monitors around the alerting one, and at past confirmed incidents, before acting
Demo outcome	3 alerts: 1 mitigated, 2 held (a lone breach and a flapping new service)
Example id	`incident_response` (`uv run grounded-decision examples incident_response`)

The decision today¶

The bot receives an alert from monitoring and can run a playbook: roll back the last deploy, shift traffic to another region, restart a pool. Either choice can be wrong, and the two mistakes cost different things.

Mistake	What it costs
Bot mitigates when nothing was wrong	A needless rollback or traffic shift, possibly overloading the other region; a human paged to undo it
Bot holds when the outage was real	Minutes of downtime while a person looks at the alert and runs the playbook by hand

The incident has a lifecycle, and the bot follows it as a small policy (the list of which events lead to which actions):

stateDiagram-v2
    [*] --> watch
    watch --> mitigating: alert
    mitigating --> paging: still_bad
    paging --> mitigating: ack
    mitigating --> resolved: clear
    resolved --> mitigating: alert
    note right of mitigating: proposes mitigate
    note right of paging: proposes page_oncall
    note right of resolved: proposes close

Each box is a state (where this incident currently is); each arrow is an event the bot receives. Arriving in a state gives a proposed action: what the policy would do next if nothing else were checked.

Where today's rule goes wrong¶

"Mitigate on every alert" is right when the alert is one symptom of a broad failure and wrong when it stands alone. The alert on its own cannot tell you which. The demo sends three alerts:

A real outage. Checkout, payments and cart are all breaching their SLOs together. Mitigating is right.
A lone breach. One latency monitor on checkout breaches; its 15 neighbours (the rest of checkout, payments, cart and inventory) are healthy. No incident confirmed by a post-mortem ever looked like this. It is usually a noisy host or a bad exporter deploy.
A flapping new service. recommendations was onboarded last week and has no dependencies in the service catalog yet. Two of its four monitors breach, two don't, and knowing one tells you nothing about the others.

The naive bot rolls back on all three.

What the system checks, in your terms¶

Before the proposed action runs, the library asks six yes/no checks. Any "no" means the decision is held. Four of them read the data you send with the alert; two check the policy itself once, when the bot starts. Three of the four are below; the fourth, support, is not measured in this example and passes with support not measured.

Check	Question in SRE terms	Data it reads	Fails when	What a hold means
Neighbour agreement	Do linked monitors tend to be in the same state, so the neighbourhood actually tells us something?	Monitor states and the links between monitors (same service, or services that call each other)	Monitors breach and recover independently of their neighbours (the flapping service)	"This neighbourhood is noise; a person should look"
Knife-edge	Is the neighbourhood's link density so close to our cut-off that one more or fewer dependency would flip the reading?	Number of monitors and links; your density cut-off (0.25)	Density sits within the window around the cut-off; the window is wide for small neighbourhoods	"The reading is unstable; a person should look"
Unusual case	Does this alert's neighbourhood look like the neighbourhoods of past confirmed incidents?	The share of neighbouring monitors that are healthy, compared with the same share for past confirmed incidents	The healthy share is higher than almost every confirmed incident's (the lone breach)	"We have never been right on an alert like this"
Policy shape	Is the policy simple enough to compile and audit?	The policy	Never in this demo	Fixed when the bot starts, not per alert
Progress	Can the lifecycle keep moving?	The policy	Never in this demo; see the warning below	Fixed when the bot starts, not per alert

What these checks do not verify

They do not check that the evidence points towards the proposed action. Neighbour agreement and the knife-edge check measure whether the neighbourhood is informative and stable, not which way it points. This example feeds the healthy share to the unusual-case check, so a lone breach in a quiet neighbourhood is caught; that is the only direction the bot sees (the support check is not measured here). Send clear for an incident whose whole neighbourhood is breaching and the bot acts close: all six checks pass (we ran it). The calibration set was built from alerts, so it says nothing about clears. If you let the bot close incidents, give clear its own evidence (for example the breaching share, calibrated on confirmed resolutions).

The progress check assumes the bot steers. Every state here has an even priority, so any run is acceptable, and the best cycle (mitigating and resolved) averages 1.5 per step, so it passes. It does not detect an incident that gets stuck in paging because nobody acks: the check reasons about which events can follow, not about events that never arrive, and waiting in a state is not a loop in the chart. Put a timer on that in your paging system.

Data you provide¶

Input	Source system	Refresh	Owner	Minimum and quality bar
Monitor states at alert time: `(monitor_id, service, is_breaching)`	Monitoring API (Prometheus, Datadog, ...)	Per alert	Observability team	Every monitor on the alerting service and its direct dependencies; read at alert time, not later
Service call pairs: `(caller, callee)`	Service catalog or distributed tracing	Daily	Platform team	New services must be registered; without entries their neighbourhood is only their own monitors
Density cut-off (`INCIDENT_DENSITY_THRESHOLD`, 0.25)	Your incident history	Quarterly	SRE lead	The density at which past real incidents and coincidental co-breaches were hardest to tell apart
Calibration set: healthy share at alert time for confirmed incidents (`CONFIRMED_INCIDENT_HEALTHY_SHARES`, 30 values)	Post-mortem database	Weekly	Incident review	At least 9 entries at the default 10% setting, or the check can never fail; only incidents a post-mortem confirmed; never include the alert being decided

The calibration values in the demo (0.00, 0.02, ..., 0.58) are synthetic. Yours come from your post-mortems, and labels arrive only after the post-mortem is written, so the set lags real incidents by days.

Worked example¶

Alert	What the bot sees	Checks	Verdict	What the on-call sees
checkout `error_rate`	12 of 16 monitors breaching, whole services together	All pass	Act `mitigate`	The playbook ran; record in the audit log
checkout `latency_p99`	1 of 16 breaching, 15 healthy neighbours	Unusual case fails: 94% of the neighbourhood is healthy, more than any confirmed incident (max 58%)	Hold, proposed `mitigate`	"Bot wanted to mitigate; held because no confirmed incident looked like this"
recommendations `error_rate`	2 of 4 breaching, no known dependencies	Neighbour agreement fails: the four monitors don't predict each other	Hold, proposed `mitigate`	"Bot wanted to mitigate; held because this neighbourhood carries no signal"

The numbers

From uv run grounded-decision examples incident_response (and --json for the summary). \(n\) is the number of monitors, \(d\) the average links per monitor, \(\lambda\) how much more often linked monitors share a state than chance (0 = no better than chance), \(d\lambda^2\) the evidence strength (must exceed 1.05), \(\hat p\) the link density, and \(p\) the unusual-case score (must exceed 0.10).

	Real outage	Lone breach	Flapping new service
\(n\) monitors, links	16, 72	16, 72	4, 6
breaching	12	1	2
\(d\), \(\lambda\)	9, 0.556	9, 0.694	3, −0.333
\(d\lambda^2\) (> 1.05)	2.78 pass	4.34 pass	0.33 fail
\(\hat p\) vs cut-off 0.25	0.60, half-window 0.209, pass	0.60, pass	1.00, half-window capped at 0.5, pass
healthy share	0.25	0.9375	0.50
\(p\) (> 0.10)	18/31 = 0.581 pass	1/31 = 0.032 fail	6/31 = 0.194 pass

Summary: naive acts 3, grounded acts 1, holds 2, blocked overconfident acts 2.

Two honest observations. The lone breach passes neighbour agreement with a higher strength (4.34) than the real outage, because its 15 healthy monitors agree strongly with each other; agreement is not direction. And \(\hat p\) is 0.60 on both checkout rows: link density depends only on which monitors and dependencies exist, not on which are breaching, so the knife-edge check cannot react to the alert itself. It only reacts to the shape and size of the neighbourhood.

What happens next¶

Act. The bot runs the playbook named in action and writes the record to the audit log. Save the new state (mitigating) against the incident.

Hold. The bot does not run the playbook. It posts the record's proposed_action and reason to the incident channel and pages the on-call at your normal alert priority. A hold is not a resolution: a real lone-breach outage (a p99 regression on checkout alone) must still reach a human quickly. Set a hold timeout in your paging tool, for example 5 minutes unacknowledged on a customer-facing service, after which it escalates like any unhandled page.

What the on-call does:

Lone breach: look at the one monitor (often a noisy host or an exporter deploy). If it is real, run the playbook by hand.
Flapping new service: decide by hand, then fix the cause: register the service's dependencies in the catalog so future alerts have a real neighbourhood.

State after a hold. The library moves the policy before it runs the checks, so a held alert still returns extras["state"] = "mitigating", although nothing ran. If you save that state, the next still_bad proposes page_oncall and the next clear proposes close, for a mitigation that never happened. The library has no separate "held" state; your service decides what to save. Two safe patterns:

Keep the old state on a hold. Save extras["state"] only when the verdict is act; on a hold keep the state you sent (watch). When the on-call decides, send the event that matches what they did.
Save it as pending. Store the returned state as "pending" next to the old one, and commit it only if the on-call runs the mitigation by hand.

The reference service (Build a decision service) saves the returned state on every decision; adapt it if you use pattern 1 or 2.

Feeding back. When the post-mortem confirms an incident, add its healthy share at alert time to the calibration set. Held alerts that turned out real are the most valuable entries: they widen what the bot recognises.

Events the policy doesn't know (a mistyped Alert) do nothing: the state stays where it was and nothing is proposed.

Tuning and expected workload¶

Density cut-off (0.25) is a policy choice from your history. See Choose thresholds and calibration.
Unusual-case level (conformal_alpha, default 0.10): lower holds fewer alerts but lets more unusual ones through.
Evidence-strength margin (reconstruction_margin, default 0.05) rarely needs changing.
Small neighbourhoods. With 6 or fewer monitors the knife-edge window is ±0.5 around the cut-off, so the check passes only when density is more than 0.5 away from it. Fully connected small groups (density 1.0) pass a 0.25 cut-off; sparser ones may always be held.
Hold rate. The demo holds 2 of 3, which tells you nothing about production. Replay a month of alerts with Estimate impact before launch to size the extra pages before you switch the bot on.

Limits, risks and compliance¶

The demo data is synthetic: four services, four monitors each.
The checks judge whether the neighbourhood supports acting at all, not which action (see the warning above).
This bot must never be the only path to a human for a customer-facing outage. Holds must page.
Saved states belong to one version of the policy. If you add a state, the names can change meaning; store a policy fingerprint with the state (State and policy versions).

Wiring it in¶

At bot start, build the module once: incident_module = build().
Weekly, refresh CONFIRMED_INCIDENT_HEALTHY_SHARES from the post-mortem database.
Per alert, pull monitor states for the alerting service and its direct dependencies, plus the call map, and decide with the saved state:

evidence = alert_neighbourhood_evidence(monitor_statuses, service_calls)
saved_state = load_incident_state(incident_id)  # None for a new incident
response = serve_one_alert(incident_module, saved_state, Event("alert"), evidence)
append_to_audit_log(response)
if response["verdict"] == "act":
    save_incident_state(incident_id, response["extras"]["state"])
    run_playbook(response["action"])  # mitigate / page_oncall / close
else:
    # Hold: keep the old state, hand the alert to a person.
    page_oncall_with(response["proposed_action"], response["reason"])

serve_one_alert wraps handle_request, which restores the saved state before deciding. The next event for the same incident (still_bad, ack, clear) passes the saved state back. load_incident_state, save_incident_state, run_playbook and page_oncall_with are yours.

Concurrency: the module is not thread-safe. handle_request restores state per request; use one module per worker or a lock.

Run it yourself¶

uv run grounded-decision examples incident_response

The output prints one block per alert. Its labels map to the words on this page:

CLI label	Meaning here
`fire`, `ACT`	act: the playbook runs
`HOLD`, `hold_for_human`	hold (the record's verdict is `abstain`)
`await_next_event`	waiting: nothing was due at this step
`cursor`, `extras['state']`	saved state
`flicker band`	knife-edge window
`bag`	calibration set
`KS`, `need > 1.05`	the evidence-strength line