Skip to content

Is the A/B lift real? (contingency tables)

An A/B test ends as a small table: conversions and non-conversions for control and for the variant. The question every launch review asks is whether the variant's lift is larger than chance would produce. This page explains the classical way to answer it by drawing random tables with the same totals, what the library's table samplers actually sample, and what the related catalog result about sampling tables claims.

It assumes you have seen a p-value. Symbols are collected in How to read these pages.

The question and the product situation

A 50/50 checkout test pools its segments into one table:

control variant
converted \(a_1\) \(a_2\)
did not \(c_1 - a_1\) \(c_2 - a_2\)
users \(c_1\) \(c_2\)

Hold the number of users in each arm (\(c_1, c_2\)) and the total number of conversions (\(r = a_1 + a_2\)) fixed. If the variant made no difference, which conversions happened to land in which arm is just chance. A permutation test asks: among random relabellings of users to arms, how often is the lift at least as large as the one observed? That random-relabelling distribution is exactly Fisher's exact test.

The set of all tables with the same row and column totals is called the fiber of the observed table.

Intuition: two different "random tables"

There are two natural ways to pick a "random table with these totals", and they are not the same:

  • Hypergeometric (Fisher, permutation): shuffle the users. Tables that can be produced by many shuffles are more likely.
  • Uniform on the fiber: every table with the right totals is equally likely, however many shuffles produce it.

Tiny example. Column totals \((2, 1)\) (two users in control, one in the variant) and one conversion in total. The fiber has two tables: the conversion is in control, or it is in the variant.

Top row (conversions) Ways to get it by shuffling Hypergeometric Uniform on fiber
(1, 0) 2 (either control user) 2/3 1/2
(0, 1) 1 1/3 1/2

The library's two-row sampler draws the hypergeometric law. Checked by sampling 30,000 tables with sample_two_row((2, 1), 1, rng=np.random.default_rng(0)) in a loop: 19,971 times (1, 0) and 10,029 times (0, 1), about 2:1. With column totals \((3, 3)\) and three conversions it gives about 1 : 9 : 9 : 1 over the four tables, the binomial-coefficient pattern \(\binom{3}{a}\binom{3}{3-a}\), not 1 : 1 : 1 : 1.

For a permutation test or Fisher's exact test, the hypergeometric law is the right null. The uniform law answers a different question (counting tables, or certain conditional tests in log-linear models).

Worked example

uv run grounded-decision examples ab_test_tables

Three experiments, each with 18 country-by-platform segments. For each one the example sums the segments into one two-by-two table, draws 2000 random tables with the same margins using sample_two_row, and computes

\[ p = \frac{1 + \#\{\text{draws with } \lvert\text{lift}\rvert \ge \lvert\text{observed lift}\rvert\}}{1 + 2000}. \]

The \(+1\) keeps the p-value away from zero, the same convention the conformal check uses.

Experiment Pooled lift p-value Decision kernel
consistent_win +9.1% 0.0030 acts: ship_variant
concentrated_win +6.7% 0.0185 holds: the variant wins only 8 of 18 segments; two carry the lift
broken_split +12.5% 0.0005 holds: users split 360/720 in a 50/50 test

All three pass "ship if p < 0.05". The decision module then checks separate evidence (whether the win is consistent across segments, and whether the randomisation split looks normal) and ships only one. A small p-value is necessary, not sufficient.

The precise statement

A contingency table with row sums \(r_1,\dots,r_m\) and column sums \(c_1,\dots,c_n\) (equal totals) is a nonnegative integer matrix with those margins; the set of all of them is the fiber. Two classical problems are to sample a table uniformly from the fiber and to count the fiber (or approximate the count within a factor \(1 \pm \varepsilon\) with high probability in polynomial time: a fully polynomial randomized approximation scheme, FPRAS).

For two rows, the hypergeometric law used above has the closed form

\[ P(\text{top row} = a_1,\dots,a_n) \;=\; \frac{\prod_{j} \binom{c_j}{a_j}}{\binom{N}{r_1}}, \qquad N = \textstyle\sum_j c_j, \]

and sequential hypergeometric draws (column by column) sample it exactly.

Where this result stands

In the wider literature: counting contingency tables exactly is

P-hard. Efficient uniform sampling and approximate counting are

known when the number of rows is a fixed constant (Dyer, Kannan and Mount) and in various special cases (for example, large margins); for arbitrary margins and dimensions, polynomial-time exact uniform sampling and an FPRAS are open problems.

In this project's math catalog (catalog result 115): an exact uniform sampler in expected polynomial bit time for arbitrary margins, an almost-uniform sampler with a worst-case polynomial time bound, and an FPRAS for counting tables with individual cell bounds, checked by a Lean statement in the companion math catalog (kept outside this repository).

What this library relies on: none of those algorithms. The pin is there so a reader can see the gap between what the catalog states and what the package runs. Status of every result: Where each result stands.

What the code does, and how it differs from the catalog result

grounded_decision.compare.tables has two samplers. Each returns a TableSample with table, method, mixer_status, and log_count (always None: the fiber is never counted).

sample_two_row(col_sums, row0, rng=None): walks the columns left to right, drawing the top-row cell of each column from a hypergeometric distribution given what is left, then fills the bottom row as the column total minus the top. This samples the hypergeometric (Fisher exact test, permutation) law exactly. It is not uniform on the fiber. method="two_row_hypergeometric", mixer_status="exact_hypergeometric". Raises ValueError if row0 is outside \([0, \sum c_j]\). Example: sample_two_row((10, 8, 6), 12, rng=np.random.default_rng(0)) returns [[4, 6, 2], [6, 2, 4]].

sample_switch(row_sums, col_sums, steps=2000, rng=None): for general tables. Fills a feasible table greedily, then makes steps random two-by-two moves (add 1 on one diagonal, subtract 1 on the other, rejecting moves that would go negative). Each move keeps the margins. The chain is symmetric, so in the long run it is uniform on the fiber; how many steps are enough is not certified. method= "switch_chain", mixer_status="switch_steps=<steps>_not_paper_fpras"; a table with one row or one column returns mixer_status="degenerate". Example: sample_switch((5, 5), (4, 3, 3), steps=20, rng=np.random.default_rng(1)) returns [[2, 3, 0], [2, 0, 3]] with switch_steps=20_not_paper_fpras.

Differences from the catalog result:

  • The two-row sampler draws a different distribution (hypergeometric, not uniform), deliberately, because that is the right null for an A/B test.
  • The switch chain is uniform only in the limit; finite runs have no error bound, unlike the catalog's samplers.
  • Nothing counts tables and nothing supports cell bounds or structural zeros.
  • Limits.max_table_cells_exact (64) and Limits.max_switch_steps (10,000) are not consulted by these helpers: the caller chooses steps, and the two-row path runs regardless of table size.

Limits and when it matters in practice

  • For a single two-by-two A/B table you can also run Fisher's exact test directly (for example scipy.stats.fisher_exact); the Monte Carlo version agrees with it up to sampling error. With 2000 draws the smallest possible p-value is \(1/2001 \approx 0.0005\), and a p-value near 0.05 has a Monte Carlo standard error of about 0.005; use more draws when the answer is close to your significance level.
  • A chi-square test is a large-sample approximation to the same question; it is fine with large counts in every cell and unreliable with small ones.
  • The table test says whether the pooled lift is larger than chance. It says nothing about whether the lift is consistent across segments or whether randomisation was broken; that is why the example passes the result to the decision module. Choosing the module's parameters is covered in Choose parameters.
  • For tables with more than two rows, the switch chain gives approximately uniform tables; run many more steps than the default and check that your statistic has stabilised across independent chains.