Test whether the difference between two conversion rates is statistically significant.
Calculated locally in your browser.
How do you calculate A/B test significance?
Use a two-proportion z-test: z = (rate B − rate A) ÷ pooled standard error, then convert z to a p-value from the normal distribution. Below 0.05 is significant at 95% confidence. The p-value is the chance of seeing this difference if the variants were identical — not the chance B is better.
Understanding your result
The p-value is the probability of seeing a difference at least this large if the two variants were in fact identical. It is not the probability that B is better, and a result that is not significant is not proof that the variants are the same — it means you do not yet have enough evidence. The confidence interval is more informative than the verdict: if it spans zero, the true effect could plausibly be a loss. The most damaging habit in practice is checking results repeatedly and stopping the moment something crosses the threshold, which inflates false positives dramatically. Decide the sample size and duration in advance, and run at least one full business cycle so weekday and weekend behaviour are both represented.
Formula and method
The z-score is (pB − pA) divided by the pooled standard error, where the pooled proportion is total conversions over total visitors. The p-value is the normal-distribution probability of a z-score at least that extreme. The confidence interval around the observed difference uses the unpooled standard error.
Assumptions and limitations
This tests a single binary metric between two variants at one point in time. It does not correct for testing several variants or metrics at once, which inflates false positives, and it assumes visitors were randomly assigned and counted once. It also cannot detect a broken test — mis-tracked conversions, uneven traffic splits or bot traffic will produce a confident answer to the wrong question.
Worked example
241 conversions from 4,820 visitors is 5.00%; 302 from 4,790 is 6.30%. That is an absolute lift of 1.30 percentage points, a relative lift of 26%, a z-score of about 2.72 and a p-value near 0.0065 — significant at 95% confidence.
How to use this tool
- Enter visitors and conversions for the control and the variant.
- Choose your confidence level, usually 95%.
- Read the p-value and the verdict.
- Check whether the confidence interval crosses zero before acting.
Common mistakes to avoid
- Stopping the test as soon as it first looks significant.
- Reading the p-value as the probability that B is better.
- Confusing absolute lift in percentage points with relative lift.
- Running for a few days and missing weekly traffic patterns.
About the A/B Test Significance Calculator
This calculator runs a two-proportion z-test on your A/B test results. It reports the conversion rate of each variant, the absolute and relative lift, the z-score, the p-value, a 95% confidence interval around the difference, and a plain verdict on whether the result is statistically significant.
Who should use this tool
Marketers, product managers and developers running split tests on landing pages, emails or features.
Benefits
- Gives the confidence interval, not just a pass or fail verdict.
- Warns when the sample is too small for the test to be reliable.
- Reports both absolute and relative lift, which are easily confused.
- Supports one- and two-tailed tests at three confidence levels.
Practical use cases
- Deciding whether a landing page variant genuinely won.
- Checking an email subject-line test.
- Validating a pricing or checkout experiment.
- Sanity-checking a result your testing tool reported.
Frequently asked questions
What does statistical significance actually mean?
That the observed difference would be unlikely to arise by chance if the variants were identical. At 95% confidence, roughly one significant result in twenty is a false positive.
How many visitors do I need?
It depends on your baseline rate and the size of the effect you want to detect. As a rough floor, aim for several hundred conversions per variant; work out the exact figure before starting with a sample size calculator.
Should I use a one-tailed or two-tailed test?
Two-tailed in almost all cases. A one-tailed test only asks whether B is better and ignores the possibility that it is worse, which makes significance easier to reach and is rarely justified.
My result is not significant. Has the test failed?
No — it means there is not enough evidence yet. Either the true difference is small, or you need more data. A wide confidence interval tells you which.
What is the difference between absolute and relative lift?
Going from 5% to 6.3% is an absolute lift of 1.3 percentage points but a relative lift of 26%. Reporting the relative figure without the baseline is how modest wins get oversold.