A client rarely asks for a p-value. The real question is simpler: what should happen now? A useful answer has four parts: the estimated effect, the plausible range around it, what that range means for the business and the next decision.
The example below is hypothetical. It shows the same test with three possible results, so the wording can be compared directly.
Agree on the decision rule before the test starts
A shop tests a checkout add-on with a fixed fee of €2,000 per month. Users are split evenly, with 20,000 in each group for one month. The main metric is the share of users who buy. In the control group, 800 users buy, which is 4.00%.
The shop gets about 40,000 users per month and earns €25 of margin per buying user. The test assumes both stay stable after launch. One percentage point is therefore 400 buyers, or €10,000 of margin per month. The fee is covered at +0.20 percentage points, because 0.20 × €10,000 = €2,000.
That break-even point matters more than zero. A real but small lift can still lose money. The rule, written down before launch, uses the 95% interval for the difference:
- Launch if the whole interval is above +0.20 points.
- Do not launch if the whole interval is below +0.20 points.
- Treat as unresolved if the interval includes +0.20 points. The default is to keep the current checkout, because the add-on has an ongoing cost.
Turn the raw counts into an effect and a range
Each row is a separate possible outcome of the test. Net value is the difference in points × €10,000, minus the €2,000 fee.
