How to Read an A/B Test

What this metric or method tells you

A test estimates how variants performed for the exposed population during the experiment. Statistical output does not repair a biased audience, broken assignment, or a metric selected after seeing results.

The useful question is not whether a number moved in isolation. Ask which audience, page, campaign, device, and time period produced the change. Keep the definition and reporting scope visible so another person can reproduce the comparison.

A practical analysis workflow

1. Write the decision first. State what you will change if the evidence points in one direction or another.
2. Check the collection. Confirm the event, date range, filters, consent behavior, and comparison period before interpreting the chart.
3. Segment carefully. Break the result down only where the segment has enough volume and a plausible reason to behave differently.
4. Look for supporting evidence. Pair the headline metric with volume, quality, and downstream outcomes.
5. Record the next step. Document the finding, limitation, owner, and date for a follow-up check.

Actions worth taking

• Choose the primary metric before starting.
• Run through normal business cycles.
• Monitor safety and quality metrics.

Start with the smallest change that can test the explanation. Preserve the earlier report or annotation so you can distinguish the effect of the change from seasonality and ordinary variation.

Common interpretation mistakes

• Stopping as soon as one day looks positive.
• Running overlapping tests on the same element without design.

A dashboard cannot remove uncertainty. Use these measures as evidence, state the limits, and avoid presenting estimated or attributed values as exact counts of individual people.

Review checklist

• The metric definition and denominator are written beside the result.
• The date range and comparison period represent similar business conditions.
• Internal, test, and known automated traffic are handled consistently.
• No personal or sensitive information appears in URLs, event names, or parameters.
• The conclusion leads to a specific decision or a clearly defined follow-up question.