A/B test launch: ship the variant or keep control¶

Your experiment has reached its planned sample size and the pooled lift is significant, so the usual launch rule says ship. Two things often make such a result wrong anyway: the lift comes from a few segments and is flat or negative elsewhere, or the randomisation was broken and the arms are not comparable. This page shows how the library keeps your significance test and adds checks on segment consistency and on the traffic split before the feature flag is flipped. It is written for experimentation data scientists and the product manager who signs off a launch; the maths is in collapsed boxes and linked pages.

At a glance


Decision	Make the variant the default (`ship_variant`) or keep control (`keep_control`).
Owner	The experiment owner and the launch review.
Today's rule	Ship if the pooled conversion lift is significant at p < 0.05.
What this adds	Before shipping, check that wins are consistent across country × platform segments and that the traffic split looks like a healthy A/A test. Otherwise hold for the experiment owner.
Demo outcome	3 experiments, all significant: 1 shipped, 2 held (one concentrated win, one broken split).
Example id	`ab_test_tables` (`uv run grounded-decision examples ab_test_tables`)

The decision today¶

A 50/50 test of a new checkout page runs until its planned sample size. The launch review computes the pooled conversion lift and its p-value and ships when \(p < 0.05\) and the lift is positive.

Mistake	What it costs
Ship a lift that is not real everywhere	The default changes for every user on the strength of a few segments; some segments get worse.
Ship on a broken split	The measured lift is an artefact (lost users, bot filtering on one arm); the launch decision has no basis.
Hold a good result	The experiment runs longer or waits for review; the win arrives later.

Where today's rule goes wrong¶

The demo has three experiments over a grid of 6 countries × 3 platforms (18 segments). All three pass the p < 0.05 bar.

Consistent win. Pooled lift +9.1 %. The variant converts better in 17 of 18 segments, with a clean 540/540 split. Shipping is right.
Concentrated win. Pooled lift +6.7 %, still significant. But the variant converts 25 of 30 users in two segments, 10 of 30 in six, and 7 of 30 (below control's 8) in the other ten. It wins only 8 of 18 segments. Shipping changes the default for sixteen segments on the strength of two.
Broken split. Pooled lift +12.5 % and the variant wins every segment. But a test configured as 50/50 put 720 users in the variant and 360 in control. This is a sample ratio mismatch: something (a redirect that drops users, bot filtering on one arm) made the arms different, and the lift cannot be trusted however small its p-value.

Today's rule ships all three.

What the system checks, in your terms¶

The pooled test still runs first and sets the event: variant_wins when the lift is positive and significant, otherwise no_clear_winner. Then the library runs six checks on the evidence; all must pass to carry out the proposed action. Two check the policy itself and pass here (see the six checks). Three of the four that read your data are below; the fourth, support, is not measured in this example and passes with support not measured:

Check	Question, in experimentation terms	Reads	Fails when	What a hold means
Neighbour agreement	Do segments that share a country or a platform tend to agree on whether the variant won?	Per-segment "variant won here" flags; segments linked when they share a country or platform	Wins are scattered, so neighbouring segments disagree about as often as they agree	"The effect is not coherent across the grid."
Knife-edge	Is the share of segments the variant won clearly away from a coin flip (50 %)?	Number of segments won, out of 18	The share is inside the uncertainty window around 50 %	"The variant wins in about half the segments."
Unusual cases	Does this test's traffic split look like the splits of recent healthy A/A tests?	Sample-ratio-mismatch statistic for this test; the same statistic from the last 30 A/A tests (the calibration set)	This test's mismatch is larger than almost all of the A/A tests'	"The randomisation looks broken; do not analyse further."

What these checks do NOT verify

Which way the segments point. The knife-edge check asks whether the share of segments won is far from 50 %, in either direction, and the agreement check asks whether neighbouring segments agree, about winning or about losing. A variant that wins big in the three US segments and loses in the other fifteen has a significant pooled lift (+9.4 %, p = 0.0025), a 3-of-18 win share far from 50 %, and coherent losses, so it acts ship_variant. (Reproduced with the shipped ab_test_tables functions via uv run python -c, setting segments 0–2 to 30 conversions and the rest to 7.) Read the segment breakdown before you ship, or add your own rule upstream (for example, send no_clear_winner unless the variant wins a majority of segments).
Segment size. "Won the segment" counts a segment of 30 users the same as one of 30,000, and ignores noise within a segment.
Whether a heterogeneous effect is bad. A win in some markets and a loss in others is a product question (ship to some segments? fix and re-test?), not something the library decides.
Peeking and multiple testing. The pooled p-value is valid once, at the planned sample size. Running this check daily and shipping the first time it acts inflates false launches, and so does testing many metrics or variants at once.
Guardrail metrics (latency, refunds, unsubscribes). Combine them with this decision using composition so the launch acts only when every guardrail also acts.

Data you provide¶

Input	Source system	Refresh	Owner	Minimum / quality bar
Per-segment counts: users and conversions by country × platform × arm (`SegmentResult` rows)	Exposure and conversion tables, `GROUP BY country, platform, arm`	Once, at the planned sample size	Experimentation platform	Same segment grid for every experiment; enough users per segment that "won" is not mostly noise.
A/A history: the mismatch statistic of recent A/A tests (`RECENT_HEALTHY_SAMPLE_RATIO_STATISTICS`, 30 in the demo)	Experimentation platform	Append after each A/A test	Experimentation platform	At least 9 at the default 10 % level, or the check can never pass.
Number of random tables for the pooled p-value (`RANDOM_TABLE_COUNT`, 2000)	Configuration	—	Experimentation DS	2000 gives p-values accurate to about ±0.005 near 0.05; raise it if decisions are often close.

launch_review_inputs in ab_test_tables.py computes the event and the evidence from these.

Worked example¶

These rows come from uv run grounded-decision examples ab_test_tables.

Experiment	Pooled test (event)	Agreement check	Knife-edge check	Unusual-cases check	Today's rule	This system
Consistent win	+9.1 %, p = 0.0030 (`variant_wins`)	Pass: neighbouring segments agree	Pass: won 17 of 18	Pass: 540/540 split	ship	act `ship_variant`
Concentrated win	+6.7 %, p = 0.0185 (`variant_wins`)	Fail: wins scattered; neighbours disagree more than agree	Fail: won 8 of 18, close to half	Pass: 540/540 split	ship	hold; proposed `ship_variant` kept on the record
Broken split	+12.5 %, p = 0.0005 (`variant_wins`)	Pass	Pass: won 18 of 18	Fail: 720/360 split, far beyond any healthy A/A test	ship	hold; proposed `ship_variant` kept on the record

The reasons tell the owner what to fix: agreement or knife-edge means "the effect is not consistent; look at the segment breakdown"; unusual cases means "the assignment is broken; fix the randomisation and restart". Over three experiments today's rule ships three times; this system ships once (blocked_overconfident_acts: 2 in the --json summary).

The numbers behind each check

Pooled p-value. The pooled result is a 2×2 table (converted or not, by arm). The example draws 2000 random tables with the same row and column totals, as if the same conversions had been shuffled between the arms at random: the null distribution of Fisher's exact test, drawn by compare.tables.sample_two_row, which reports mixer_status="exact_hypergeometric". The p-value counts how often a random table shows a lift at least as large: \(p = (1 + \#\{|\text{random lift}| \ge |\text{observed lift}|\}) / (1 + 2000)\).

Segment checks. Every segment is linked to 7 others (5 other countries on the same platform, 2 other platforms in the same country), so \(d = 7\). \(\lambda\) is how much more often linked segments agree on "variant won" than chance; negative means they disagree more than chance. Evidence strength \(d\lambda^2\) must be above 1.05. The knife-edge half-width at \(n = 18\) segments is 0.193 around 0.5.

	Consistent	Concentrated	Broken split
segments won \(\hat p\)	17/18 = 0.944	8/18 = 0.444	18/18 = 1.0
distance from 0.5 vs 0.193	0.444 (pass)	0.056 (fail)	0.5 (pass)
agreement \(\lambda\)	0.778	−0.206	1.0
evidence strength \(d\lambda^2\)	4.235 (pass)	0.298 (fail)	7.0 (pass)
mismatch statistic	0	0	120
unusualness p-value	1.0 (pass)	1.0 (pass)	0.032 (fail)

The mismatch statistic is \((v - c)^2 / (v + c)\) for \(v\) variant and \(c\) control users: \(360^2 / 1080 = 120\). The 30 A/A statistics run from 0 to 3.77. The unusualness p-value is \((1 + \#\{\text{A/A statistics} \ge 120\}) / (1 + 30) = 1/31 \approx 0.032\).

Why compare with A/A tests instead of the standard test? The usual sample-ratio-mismatch check is a chi-square test with one degree of freedom on the same statistic; for 120 it gives \(p \approx 6 \times 10^{-28}\) (computed with uv run python -c). The demo routes the split through the unusual-cases check instead, to show that check on a familiar quantity. The price is a floor: with 30 A/A tests the smallest possible p-value is \(1/31\), so this check can say "more extreme than every A/A test we have" but never "astronomically unlikely". In production you can run the chi-square test yourself and send no_clear_winner (or refuse to decide) when it fails; the comparison with your own A/A history remains useful because it reflects your platform's real behaviour.

Full derivations: neighbour agreement, knife-edge, unusual cases, random tables with fixed totals.

What happens next¶

Verdict	Who is told	What they see	Time limit	If nobody acts	Feedback
Act `ship_variant`	Experiment owner, feature-flag service	`action=ship_variant`, full record	At the launch review	n/a	Monitor post-launch metrics against the experiment's estimate.
Act `keep_control`	Experiment owner	`action=keep_control`	At the launch review	n/a	—
Hold, consistency	Experiment owner	Proposed `ship_variant`, failed agreement or knife-edge check	Before the next launch review	Do not flip the flag. Keep control.	The owner decides whether to ship to some segments, iterate, or stop. Re-running the same test longer and re-checking is peeking (see above).
Hold, broken split	Experiment owner and platform team	Proposed `ship_variant`, failed unusual-cases check	Immediately	Do not flip the flag, and do not analyse this data further.	Fix the randomisation, restart the test. Do not add this test to the A/A history.

Each experiment is decided once; there is no state carried between experiments.

Tuning and expected workload¶

Setting	Policy or data?	How to choose	Too tight	Too loose
Significance level (0.05)	Policy	Your existing launch bar	Fewer `variant_wins` events	More false wins reach the checks
Coin-flip line (0.5)	Policy	Keep 0.5 unless you require a supermajority	—	—
Segment grid	Data	Dimensions where you would act differently (country, platform)	Many tiny segments: "won" becomes noise	Few segments: the knife-edge window is wide (half-width 0.5 up to about 6 segments)
Unusualness level α (0.10) and A/A history size (30)	Policy / data	More A/A tests lower the p-value floor \(1/(n+1)\)	More holds	Broken splits slip through

Expected holds. Broken splits should be rare on a healthy platform; if they are not, that is the finding. Consistency holds depend on how heterogeneous your effects are: replay the last quarter's launches through the checks before switching them on: Estimate impact before launch.

Limits, risks and compliance¶

Direction is not checked (see the warning above): a variant that loses in most segments but wins big in a few can ship.
Segment wins are unweighted and ignore within-segment noise.
The p-value floor of the A/A comparison (1/31 here), as above.
The pooled p-value comes from 2000 random draws, so it carries sampling error of about ±0.005 near 0.05.
Synthetic data. Segment counts are hand-built to show one ship and two holds.

Wiring it in¶

Build once with build(): variant_wins proposes ship_variant, no_clear_winner proposes keep_control.
Query per-segment counts with a GROUP BY country, platform, arm over exposure and conversion tables, as SegmentResult rows.
Maintain the A/A history: store the mismatch statistic of every A/A test and keep the most recent ones as RECENT_HEALTHY_SAMPLE_RATIO_STATISTICS.
At the planned sample size, decide once:

import numpy as np
from grounded_decision.examples.ab_test_tables import build, launch_review_inputs

module = build()
event, evidence = launch_review_inputs(segments, np.random.default_rng(0))
record = module.decide(event, evidence)
if record.verdict == "act":
    set_feature_flag(record.action)  # "ship_variant" or "keep_control"
else:
    notify_experiment_owner(record.proposed_action, record.reason)

set_feature_flag and notify_experiment_owner are your own functions. Persist record.to_dict() with the launch review; the event payload carries pooled_lift and pooled_p_value, so the record shows today's evidence next to the verdict.

Run it yourself¶

uv run grounded-decision examples ab_test_tables
uv run grounded-decision examples ab_test_tables --json

How to read the report: Example library: reading a report. HOLD and hold_for_human in the output are this page's "hold" (the record's verdict field says abstain). FLICKER marks evidence inside the knife-edge window; WEIRD marks a failed unusual-cases check.

Part	Result	Math catalog number	Trust level
Pooled p-value	Fisher exact / permutation null on 2×2 tables with fixed totals, sampled exactly	115	standard statistics; the two-row sampler is exact
Neighbour agreement	Kesten–Stigum reconstruction	229	proven (`lean`) above the line; holding at or below it is policy
Knife-edge	Friedgut–Kalai sharp thresholds	186	practical (`operational`)
Unusual cases	Split conformal prediction	—	published
Policy shape and progress	Star height; parity and mean-payoff games	134, 104	proven (`lean`)