“Find out why conversion dropped” sounds like a clear analytics task. It still leaves important choices open. Which conversion, among whom, over what period, and compared with what? If those choices stay hidden, a dashboard can produce a precise answer to the wrong question. Write a short analysis brief first, then use it to choose the data and method.
Turn the request into a question someone can answer
Start with the choice that prompted the request. A shop owner might need to decide whether to investigate a recent checkout release before spending more on acquisition. That gives the analysis a purpose. “Did checkout completion fall within comparable device groups after the release?” is a useful first question. “Did the release cause the fall?” is a stronger question that needs stronger evidence.
Write the question without assuming the suspected explanation is true. Record the decision owner, the action available and the deadline. Agree on what evidence would justify a technical investigation, a traffic review or no immediate change. GOV.UK’s guidance on capturing research questions recommends agreeing what the team needs to learn before choosing research activities. Apply that order to analytics too.
Fix the population, outcome and comparison
Define the unit before calculating a rate. For this investigation, use unique checkout attempts with a real cart, excluding internal tests. Define completion as a linked paid order within 24 hours. Count an attempt once even if the confirmation page loads twice. The 24-hour window is an illustrative rule; choose a window that fits the payment process and state how late payments are handled.
Choose two complete periods with comparable weekday coverage. Exclude the deployment interval if customers saw mixed versions. Let every attempt reach the end of its follow-up window before comparing completion. Record time zone, campaign changes, stock availability and any change in data collection. A period with missing payment updates is not a fair comparison, even if the dashboard shows a full calendar week.
