Plan an A/B test that can give a usable answer: write the hypothesis, pick one metric, calculate sample size, fix the split and check for sample ratio mismatch.
8 min readBranislav Mateas
Copy post as Markdown
On this page
An A/B test can run for weeks and still give no usable answer. This happens when the metric, the sample or the stopping rule is chosen after the data arrives. The fix is a short plan, written before launch, that states what will be measured, how many users are needed and which result leads to which decision.
This article builds such a plan for one hypothetical e-shop. All its traffic, rates and costs are invented for illustration. The calculations were done by hand and shown so they can be checked.
Write a hypothesis that can fail
A useful hypothesis names the change, the audience, the metric, the minimum effect and the reason. For the example shop:
Showing delivery cost and delivery date on the product page will raise the share of product page viewers who buy within 7 days from 4.0% to at least 4.6%, because some shoppers now leave at checkout when they first see the delivery cost.
The reason should come from evidence. Here, that would be a visible drop at the delivery step of the checkout funnel.
"The new page will perform better" is not testable, because it does not say better on what or by how much. A simple check is to ask which result would make the team drop the idea. If nobody can answer, the hypothesis is not ready.
Define one primary metric
The primary metric decides the test. Secondary metrics help explain the result, but they do not get to rescue it. Write the definition precisely enough that two analysts would get the same number:
Unit: a user, assigned at their first product page view.
Denominator: all assigned users in that group.
Numerator: assigned users with at least one purchase within 7 days of assignment.
The assignment unit and the metric unit should match. If users are assigned but the metric is counted per session, several sessions from one user are treated as independent observations, and the uncertainty is understated.
Add one or two guardrail metrics that can block a rollout without being able to declare a win. For this change, cancelled orders and revenue per purchaser would fit.
Calculate the sample size before launch
Start with the smallest effect worth acting on. Suppose building and maintaining the feature costs €35,000 in the first year, an extra order earns €20 in margin, and 6,000 new eligible users arrive each week.
Orders needed to break even: 35,000 / 20 = 1,750 a year.
Per week: 1,750 / 52 = 33.65.
As a rate: 33.65 / 6,000 = 0.56 percentage points.
The team rounds this up to 0.6 points, which means moving from 4.0% to 4.6%. It plans a two-sided test at a 5% significance level with 80% power. Power is the chance of detecting the effect if it really is that large.
The per-group estimate uses a normal approximation for two independent proportions:
The team plans 18,000 users per group, or 36,000 in total. At 6,000 a week, that takes 6 weeks.
Evan Miller's rule of thumb offers a quick check: n = 16 × σ² / δ², where σ² is p(1 − p) for a conversion rate and δ is the difference. That gives 16 × 0.0384 / 0.000036 = 17,067. It is about 5% lower because it uses only the baseline variance, so the two results agree in scale. For revenue per user, the proportion formula does not apply, and σ² has to come from historical data.
Running the formula in reverse shows what a shorter test could detect:
Weeks
Users per group
Smallest detectable lift
Relative lift
2
6,000
1.06 points (to 5.06%)
26.6%
3
9,000
0.86 points (to 4.86%)
21.5%
4
12,000
0.74 points (to 4.74%)
18.5%
6
18,000
0.60 points (to 4.60%)
15.0%
This table supports a real decision. If the shop can only run for 3 weeks, the test is unlikely to detect a lift of 0.6 points. The options are a bolder change, a higher-traffic page or no test.
Weekly counts should come from unique users over a past period of the same length, because returning users make totals grow more slowly than a weekly figure suggests.
Fix the split and the stopping conditions
An equal split reaches the target fastest. A smaller share for the new version reduces exposure but costs time. With 20% of users on the new version, the same formula gives 7.849 × (0.0384 / 0.8 + 0.043884 / 0.2) / 0.000036 = 58,305 users in total: 46,644 in control and 11,661 in the new version. That takes 10 weeks instead of 6.
The plan should also state these conditions:
Assignment: random, stable for each browser, and logged at the same point in both groups, including control.
Enrolment: exactly 6 whole weeks, so both groups cover every weekday equally.
Follow-up: 7 more days, so every user gets the full 7-day purchase window. The test takes 7 weeks in total.
Early stopping: allowed for a technical fault, without declaring a winner.
Exclusions: written in advance and applied identically to both groups.
Stopping as soon as the result looks significant breaks the plan. In Miller's example, a change with no effect, checked repeatedly with stopping at 5% significance or after 150 observations, gives a false positive rate of 26.1%. A team that needs to stop early should choose a sequential method before launch, not improvise one during the test.
Check the sample ratio before reading the result
A sample ratio mismatch means the observed split differs from the planned one by more than chance would explain. The KDD 2019 paper on diagnosing these mismatches warns that ignoring one can make a harmful change look beneficial, or the reverse.
The check is a chi-square goodness-of-fit test on user counts. For each group, calculate (observed − expected)² / expected, then add the results. Compare the sum with the critical value for 1 degree of freedom, which is 6.635 at the 0.01 level. Lukas Vermeer's SRM documentation says his retired checker used the same 0.01 threshold.
With 36,000 users and a planned 50/50 split, each group expects 18,000:
Group A
Group B
Calculation
Chi-square
Reading
18,085
17,915
2 × 85² / 18,000
0.80
No mismatch flagged
18,412
17,588
2 × 412² / 18,000
18.86
Mismatch
The second split is only 51.1% to 48.9%, which looks harmless on a dashboard. At this sample size, it is not. For a planned 80/20 split, the expected counts would be 28,800 and 7,200 instead.
A spreadsheet can return the probability with CHISQ.DIST.RT. This formula was not run for this article:
=CHISQ.DIST.RT(18.86, 1)
A flagged mismatch stops the evaluation. Vermeer's documentation says the results then cannot be analysed without additional assumptions and changed calculations. Instead, break the counts down by day, browser, device and entry page to see where the gap appears. His taxonomy of causes includes redirecting only some variants, a variant changing performance, bot removal, ad blockers and a wrong filter condition.
Consent can act in a similar way. If analytics runs only after consent, and one version changes how often visitors consent or how often the tag fires, the logged counts can diverge. That link is an inference from how consent-gated measurement works, so it needs checking in the actual setup.
The check reads only group counts, not the primary metric, so running it during the test does not create the peeking problem above. Running it in the first week can save the remaining weeks.
Write the decision rules before launch
The plan ends by mapping each outcome to an action. For the example shop:
Outcome
Action
Sample ratio mismatch or failed tracking check
Do not evaluate. Diagnose, fix and restart.
Guardrail clearly worse
Do not ship. Investigate the cause.
Significant lift, estimate at or above 0.56 points
Ship. The estimate covers the first-year cost.
Significant lift, estimate below 0.56 points
Do not ship at this cost. Look for a cheaper build.
No significant difference
Keep the current page. The result does not prove zero effect, and a lift smaller than 0.6 points may have gone undetected.
These rules make the test possible to evaluate because they were fixed before anyone saw the numbers. Once the data is in, the analyst only has to check which row applies.
