A/B Test Calculator
Check statistical significance between two A/B test variants using a two-proportion z-test.
Variant A (control)
Conversion rate: 10.00%
Variant B (challenger)
Conversion rate: 12.00%
Results
How the A/B test significance calculation works
This calculator runs a two-proportion z-test to determine whether the difference in conversion rates between Variant A and Variant B is statistically significant, or could simply be due to random chance.
First, a pooled conversion rate is calculated by combining conversions and visitors from both variants: p = (conversions_A + conversions_B) / (visitors_A + visitors_B). This pooled rate is used under the null hypothesis that both variants truly convert at the same rate.
The standard error of the difference is SE = sqrt(p * (1 - p) * (1/n_A + 1/n_B)), and the z-score is z = (rate_B - rate_A) / SE. The z-score is converted to a p-value using the standard normal cumulative distribution function, and confidence is reported as (1 - p-value) * 100%. A result is generally considered statistically significant once confidence reaches 95% or higher — but larger sample sizes always give more reliable results than smaller ones, so treat a low-traffic test's result with caution even if the confidence number looks high.
Built and maintained by Meet Shah · Last updated
What this tool is used for
- Testing whether two variants' conversion rates actually differ.
- Getting a p-value and a confidence level for a completed test.
- Checking whether a test has enough data to conclude anything.
- Comparing a result against a significance threshold.
- Producing the z-score and p-value a test write-up has to state.
Frequently Asked Questions
- What does statistical significance mean here?
- That a difference this large would be unlikely if the variants were genuinely identical. It is a statement about the data given no effect — not the probability that B is better, which is the reading almost everyone applies to it.
- Why does peeking at results invalidate the test?
- Because checking repeatedly and stopping when significance appears inflates the false-positive rate dramatically — over 20% for frequent peeking rather than the nominal 5%. Fixing the sample size in advance, or using a sequential method designed for it, is the fix.
- How large a sample is needed?
- It depends on the baseline rate and the effect you want to detect, and the relationship is brutal: halving the detectable effect quadruples the sample. Detecting a 1% relative lift on a 2% conversion rate needs tens of thousands per variant.
- What is the multiple comparisons problem?
- Testing many variants or many metrics raises the chance that one crosses significance by luck. At 5% each, twenty comparisons expect one false positive — which is why a test with a dozen secondary metrics almost always finds something.
- Should I stop a test early if it looks decisive?
- Only under a design that permits it. Early results are dominated by noise and by whichever users arrived first, and novelty effects fade — a lift that is real at day two frequently disappears by day fourteen.
Common errors and gotchas
- Stopping the test as soon as it looks significant, which inflates the false-positive rate dramatically.
- Running many variants and reporting the winner without correcting for multiplicity.
- Reading significance as importance, when a tiny effect can be significant at scale.
- Ignoring the sample-size calculation, so the test was never able to detect the effect sought.
- Treating a non-significant result as proof of no difference.