Correlation, Variation and Uncertainty in Everyday Analytics
A change or a correlation between two metrics does not show a cause. Worked examples show how to check week-to-week noise, uncertainty and shared causes.
7 min readBranislav Mateas
Copy post as Markdown
On this page
A hypothetical shop rewrites its checkout copy. The next week, orders rise 18%. The weekly report says the new copy increased orders.
That sentence makes a causal claim, and one before-and-after number cannot support it. Three questions come first. Is the change larger than an ordinary week's movement? How uncertain is the estimate? What else could explain it? All numbers in this article are invented for illustration and were calculated by hand.
Start by describing the change precisely
Here is the shop's data. Each week has exactly 5,000 sessions, which keeps the arithmetic simple. The copy changed at the start of week 9.
From week 8 to week 9, orders rose by 35. That is 35 / 195 = 17.9%. The conversion rate rose from 3.9% to 4.6%, which is 0.7 percentage points.
Both are correct, but they sound very different. A percentage change is relative to the starting value. A percentage point change is a plain subtraction of two rates. A report should name which one it uses.
The baseline matters too. Week 8 was slightly below the eight-week average of 4.0%. Against that average, the rise is 0.6 points.
Measure what an ordinary week looks like
An average hides spread, so the next step is to measure it. Weeks 1 to 8 contain 1,600 orders from 40,000 sessions, which gives a mean of 4.0%.
The differences from that mean, in points, are minus 0.6, 0.4, 0.1, minus 0.3, 0.6, 0.2, minus 0.3 and minus 0.1. Their squares sum to 1.12. Dividing by 7 (the number of weeks minus one) gives a variance of 0.16, so the standard deviation is 0.4 points.
A prediction interval estimates where one new week could land. For eight weeks at 95% confidence, the t value with 7 degrees of freedom is 2.365:
Week 9 sits comfortably inside that range. The table also shows something simpler: week 5 reached the same 230 orders with no copy change at all.
This does not prove the copy did nothing. It shows that one week of data cannot separate a copy effect from ordinary movement.
The method has limits. NIST lists independent observations and roughly normal data as assumptions, and it warns that the method is less robust with small samples and strong skew. A trend, a seasonal peak or a promotion in the baseline would break the first assumption.
Put a range around the difference
A confidence interval answers a narrower question: given the session counts, how much could this difference move through sampling variation alone? OpenIntro Statistics gives the standard error for a difference of two proportions. Applied to weeks 8 and 9:
SE = √(0.039 × 0.961 / 5,000 + 0.046 × 0.954 / 5,000)
= √(0.0000162726)
= 0.00403, or 0.40 points
95% interval = 0.7 ± 1.96 × 0.40
= minus 0.1 to 1.5 points
On 5,000 sessions, that range runs from about 5 fewer orders to about 75 more. The single figure of 35 extra orders looks far more certain than this range allows.
Zero is inside the interval, so the data does not rule out no change. It also does not show that the effect is zero. A true effect of one full point would fit this data as well.
The formula assumes independent samples. A week of site traffic is not a random sample, and one person can return in several sessions. The interval is therefore best read as a minimum estimate of the uncertainty. That reading is a judgement about web data, not a result of the formula. In this example, sampling alone predicts a weekly standard error of about 0.28 points at a 4% rate, while the observed spread is 0.4 points.
Even an interval that excludes zero would say nothing about the cause. A sale, a new traffic source or a tracking change could produce the same difference.
A strong correlation can come from a shared cause
The same caution applies to two metrics that move together. Here is a second hypothetical shop, with weekly paid media spend and orders across two periods.
Period
Spend (EUR)
Orders
Normal
1,000
198
Normal
1,100
204
Normal
1,200
198
Sale season
2,000
398
Sale season
2,100
404
Sale season
2,200
398
With spend in hundreds of euros, the means are 16 and 300. The Pearson correlation is:
A value of 0.99 looks like proof that spend drives orders. Now split the data by period. In the normal weeks, the differences from the means are minus 1, 0 and 1 for spend, and minus 2, 4 and minus 2 for orders. Their products sum to 2 + 0 − 2 = 0, so the correlation is 0. Sale season gives the same result.
Most of the 0.99 comes from the gap between the two periods. Sale season may lift both numbers because demand rises and the team also raises its budget. That is what a confounder does: it influences both variables and makes it hard to tell which one causes changes in the other. The direction can also run backwards, when a team sets budgets from expected demand.
The split does not prove that ads have no effect. Three points per period and a narrow spend range are too little evidence for that. It shows only that this correlation cannot answer the question.
Two more limits are worth knowing. Correlation measures linear association, so it misses curved relationships. Metrics that both trend upward can also look related for that reason alone, which Hyndman and Athanasopoulos call spurious regression. Their example pairs Australian air passenger traffic with rice production in Guinea.
A correlated metric can still help forecasting without being a cause. That helps planning, but it does not predict what happens when someone changes the metric. That last step is an inference from the source's distinction between forecasting and causation.
Match the wording to the evidence
The strength of the claim should follow from the checks completed.
What the data shows
What the report can say
Useful next step
A change inside the usual range
"Within normal weekly variation. No effect can be claimed."
Collect more weeks and compare the range again.
A change outside the usual range
"Unusual change. Cause not yet known."
List what else changed: campaigns, prices, stock, season, consent settings and tracking.
Two metrics move together
"These moved together in this period."
Split by period or segment, and compare with the trend removed.
The pattern survives the splits and the alternatives are checked
"Consistent with an effect. Other explanations remain possible."
Run a test if the decision is costly or hard to reverse.
A randomised test with a range
"Estimated effect of X, with a plausible range of Y to Z, for this audience and period."
Decide using the whole range, not only the midpoint.
The last row is different in kind. Random assignment lets a test estimate a direct effect without worrying about hidden variables that differ between groups. Without one, the useful question is still how many orders would have happened anyway, and any answer stays an estimate.
For the checkout example, the report line can be short: "Orders rose from 195 to 230 in the first week with the new copy. A week like this also occurred before the change, so this is not yet evidence of an effect." The next decision is then clear: keep collecting weeks, or split traffic and test the copy.
