Canary latency: is the mean an honest summary?¶

A team canaries a new request-routing policy, and the dashboard shows mean latency went down. Shipping on that number is safe only if the whole latency distribution moved together; if the new policy speeds up most requests while sending a few to an overloaded backend, the mean can drop while the p99, which users feel and SLOs are written on, gets worse. This page shows a helper that measures how far the whole distribution moved, what it can and cannot tell you, and how to wire it into a rollout gate. It assumes you know canaries and percentiles, not optimal transport.

At a glance


Question	Did the canary change the latency distribution by so little that the mean delta tells the whole story?
Owner	Performance engineers and experimentation data scientists; the SRE who owns the p99 SLO.
Today's habit	Ship when the canary's mean beats control.
What this adds	`compare_outcome_laws`: one distance between the two distributions, compared with an allowance you set.
Demo outcome	Canary A (every request 3 ms faster): comparable. Canary B (mean 19 ms better, p99 22 ms worse): not comparable, blocked.
Kind of example	A helper, not a decision: there is no act/abstain verdict.
Example id	`outcome_shift` (`uv run grounded-decision examples outcome_shift`)

For a review slide: "Both canaries lowered mean latency; only one kept the shape of the distribution. The gate blocked the one whose p99 got worse, and it would also block a large uniform speed-up, so a block means 'look at the quantiles', not 'regression'."

The question today¶

Two arms of a canary each serve 500 requests. Control has mean latency about 100 ms. The rollout rule compares means and ships if the canary is faster. What the rule cannot see is whether "faster" applies to every request or only to most of them.

Where today's habit goes wrong¶

The demo has two canaries, and the mean rule ships both:

Canary A takes every control request and makes it about 3 ms faster (plus 0.2 ms of jitter). The whole distribution slides left. Mean 99.5 → 96.5 ms, p99 124.6 → 121.4 ms. Shipping is right.
Canary B is faster for most requests but sends some to an overloaded backend; its spread is three times control's. Mean 99.5 → 80.9 ms (19 ms better), p99 124.6 → 146.5 ms (22 ms worse). Shipping on the mean trades a better average for a broken SLO.

graph LR
  C[control: mean 99.5, p99 124.6] -->|every request 3 ms faster| A[canary A: mean 96.5, p99 121.4]
  C -->|most faster, tail slower| B[canary B: mean 80.9, p99 146.5]

What the library will and will not claim¶

The helper lines up the two arms' sorted latencies (fastest with fastest, median with median, slowest with slowest) and takes the root-mean-square gap between matched pairs, in milliseconds. This is the Wasserstein-2 distance, \(W_2\). It then compares \(W_2\) with an allowance you set. Inside the allowance, the result is maps_comparable=True.

It says	It does not say
Every quantile of the canary is, on average, within about \(W_2\) ms of control's.	That the canary is better or worse. \(W_2\) has no direction.
When `True`: the change is small enough that the mean delta is a fair one-number summary.	When `False`: that the shape changed. A large uniform speed-up also fails (see below).
One number per metric, exact for samples of any sizes.	Anything about several metrics at once. It is one-dimensional.

A block is not always a regression

\(W_2\) counts a pure shift as distance. If canary A had been 10 ms faster on every request, \(W_2\) would be exactly 10 ms, above the 6.25 ms allowance, and the helper would return maps_comparable=False for a change that is strictly good (checked with compare_outcome_laws(control, control - 10, tau=0.5, scale=50)). Treat False as "do not decide on the mean alone; look at the quantiles". If you want a test of shape only, compare mean-centred samples (subtract each arm's mean first): for canary B the centred \(W_2\) is 20.0 ms, for canary A it is 0.11 ms.

The standard tool for this question, which a data scientist will reach for first, is to compare quantiles directly: p50, p90 and p99 deltas with bootstrap confidence intervals. In the demo the deltas are −2.9 / −2.9 / −3.2 ms for canary A and −18.3 / +8.5 / +21.9 ms for canary B. \(W_2\) adds one summary number that weighs every quantile, which is convenient as an automatic gate; the quantile table is what a human should read when the gate blocks.

Data you provide¶

Input	Source	Notes
Control latencies	Tracing or load-balancer logs	Same time window as the canary. Arms may have different sizes; every sample counts.
Canary latencies	Same
`tau` and `scale`	Chosen once, from A/A tests and past rollouts	Together they set one number: the allowance in ms (see Tuning).

Worked example¶

Canary	Mean (ms)	p99 (ms)	\(W_2\) (ms)	Allowance (ms)	Result
A	99.5 → 96.5	124.6 → 121.4	3.000	6.25	comparable: mean rule decides
B	99.5 → 80.9	124.6 → 146.5	27.29	6.25	not comparable: block, review

Canary A's quantiles each moved about 3 ms, so \(W_2\) is 3 ms. Canary B's fast requests got faster and its slow requests got slower; the gaps add up to 27.29 ms.

The numbers and where the allowance comes from (from uv run grounded-decision examples outcome_shift)

For one-dimensional samples with sorted quantile functions \(F^{-1}\) and \(G^{-1}\),

\[ W_2 = \Big( \int_0^1 \big(F^{-1}(q) - G^{-1}(q)\big)^2 \, dq \Big)^{1/2}. \]

The helper compare_outcome_laws(a, b, tau=..., scale=...) returns maps_comparable = W2 <= scale * tau**3. With tau = 0.5 (MAP_ERROR_TOLERANCE) and scale = 50 ms (LATENCY_SCALE_MS) the allowance is \(50 \times 0.5^3 = 6.25\) ms.

Where the cube comes from: the math catalog contains a result on how stable optimal transport maps (the rule that moves one distribution onto another) are when the target distribution moves. It says the map error is at most \(C\,W_2^{1/3}\), and no larger exponent works in general. Keeping the map error below \(\tau\) then needs \(W_2 \le \mathrm{scale}\cdot\tau^3\), with the unknown constant \(C\) folded into scale.

Two honest remarks for this use:

In one dimension the transport map is the quantile function, so the map error is \(W_2\). The cube-root bound is then loose, and the gate reduces to a plain threshold on \(W_2\).
scale and tau are not separate controls: only their combination \(\mathrm{scale}\cdot\tau^3\) matters. Once you tune the allowance from A/A data, the exponent no longer affects any decision.

In the wider literature, quantitative stability of optimal transport maps is an active research area and the exponent depends on the assumptions. In this project's catalog: one-third Hölder stability, shown sharp, for maps from the uniform measure on a convex body in dimension 2 or more, with a Lean-checked statement (catalog result 374). See the theory page.

What your service does with the result¶

maps_comparable=True: let your existing mean-based rule decide. The helper has only said the mean is a fair summary; it has not said the canary is better.
maps_comparable=False: block the automatic rollout and put the quantile table (p50, p90, p99, p99.9, with confidence intervals) in front of a person. If it ships after review, treat it as a new baseline: refresh SLO baselines and any calibration sets built on the old latencies.
Attach the result dict (w2, allowed_w2_for_map_error, tau, maps_comparable) to the rollout ticket.

Tuning and monitoring¶

Choose the allowance from your own history:

Run the helper on A/A pairs (control against control on different days). Their \(W_2\) is your noise floor.
Run it on a few past rollouts you know were bad.
Pick an allowance comfortably above the A/A values and below the bad ones, then set scale and tau to produce it (for example keep tau = 0.5 and set scale = allowance / 0.125).

Too tight an allowance blocks ordinary day-to-day noise and every meaningful speed-up; too loose lets a tail regression through. If your releases often shift latency uniformly, gate on mean-centred samples instead and judge the shift with the mean rule.

Limits and risks¶

One metric at a time. Run it per metric (latency, errors per request, revenue per session); do not flatten several metrics into one array.
No direction: a block does not mean "worse".
No sampling uncertainty: \(W_2\) on 500 requests has noise of its own. Use the A/A runs to see how much.
The demo arms are synthetic normal distributions with a fixed seed.

Wiring it in¶

from grounded_decision.compare.ot_stability import compare_outcome_laws

comparison = compare_outcome_laws(
    control_sample,
    canary_sample,
    tau=MAP_ERROR_TOLERANCE,
    scale=LATENCY_SCALE_MS,
)
attach_to_rollout_ticket(comparison)  # your function
if comparison["maps_comparable"]:
    decide_on_mean(control_sample, canary_sample)  # your existing rule
else:
    block_rollout_and_request_review(comparison)  # your function

This helper sits beside the decision path: it produces no decision record. If you want a decision module to hold when the gate blocks, pass the bit in as part of your own policy (see Compare outcomes and features).

Run it yourself¶

uv run grounded-decision examples outcome_shift

The output prints both arms' mean and p99, the allowance, then one block per canary: the inputs, what the mean rule would do (NAIVE), \(W_2\) against the allowance (DECISION), maps_comparable (OUTPUT), and what the rollout gate does next. HOLD in that output means the automatic rollout is blocked. Add --json for the raw result dicts.