A variant converts at 5.6% and the control at 5.0%. The testing tool reports p = 0.058. Someone asks whether that means a 94.2% chance that the variant wins. It does not. This article works through that result by hand, so the number has a clear meaning before anyone acts on it.
What the null hypothesis and the p-value actually claim
An A/B test compares two statements about the underlying conversion rates:
- Null hypothesis: the two versions have the same conversion rate.
- Alternative hypothesis: the rates differ, or differ in one stated direction.
The test does not try to prove the alternative. It assumes the null hypothesis and asks how surprising the data would be under that assumption.
The American Statistical Association statement gives an informal definition. A p-value is the probability, under a stated statistical model, that a summary of the data would be equal to or more extreme than the value observed.
The words "under a stated statistical model" matter. A p-value is calculated as if the null hypothesis and the other assumptions were true, so it cannot also report the probability that they are true.
The significance level, written as α, is the cut-off chosen before the test. NIST's engineering statistics handbook explains that at α = 0.05, a true null hypothesis is wrongly rejected 5% of the time. It lists 0.1, 0.05 and 0.01 as common values and calls the choice "somewhat arbitrary". So α sets an error rate you accept. It says nothing about how large or valuable an effect is.
