Did your testactually win?
A higher number isn't a winner, it might just be noise. Enter the visitors and conversions for your control and variant to get both rates, the real lift, the p value, and a straight yes or no on statistical significance.
Your two variations
The verdict
Control rate (A)
5.0%
Variant rate (B)
6.5%
Relative lift
30.0%
Confidence
95.8%
Conversion rate, head to head
p-value 0.0416 · z-score 2.04
Sensitivity · would it hold at a stricter bar?
Significance at each confidence level
| Confidence | Needs p ≤ | Verdict |
|---|---|---|
| 90% | 0.10 | Significant |
| 95% | 0.05 | Significant |
| 99% | 0.01 | Not yet |
A result that's significant at 90% but not 95% is a coin-flip dressed up as a winner. The stricter the bar it clears, the more confidently you can ship it.
Your move
Testing the right things? Let's make sure.
A 5.0% vs B 6.5% (+30.0% lift) — significant at 95%.
Plenty of teams run statistically clean tests on changes that were never going to move the needle. Bring me your funnel and I'll tell you what's actually worth testing first, free, on a 30 minute call.
Plain English
The difference between a win and a coin flip.
Statistical significance answers one question: is the difference between your two variations real, or could it have happened by chance? Every test shows some gap between A and B. Significance tells you whether that gap is signal you can act on or random variation you'd be foolish to ship.
The engine is a two proportion z test. It compares the conversion rates of control and variant, accounts for how many people saw each, and produces a p value, the probability of seeing a gap this big if the two were actually identical. A small p value means chance is an unlikely explanation, so the difference is probably real.
The cost of getting this wrong is brutal: call a winner too early on a fluke and you'll roll out a 'better' page that quietly performs worse, then wonder why revenue slipped. This calculator gives you the rates, the lift, the p value, and the confidence level so you ship on evidence, not hope.
The formula
z = (pᴮ − pᴬ) ÷ √[ p̄(1−p̄)(1/nᴬ + 1/nᴮ) ] · significant when p value ≤ 1 − confidence
A: 100/2,000 = 5.0%. B: 130/2,000 = 6.5%. That's a +30% relative lift with a p value near 0.04, significant at 95%, so B is a real winner.
How to read the confidence level
Confidence is just 1 minus the p value, how sure you can be the result isn't chance. The conventions most teams hold themselves to:
| Confidence | p value | What it means |
|---|---|---|
| 90% | 0.10 | Loose, useful for early reads |
| 95% | 0.05 | Standard bar for shipping a winner |
| 99% | 0.01 | Strict, high stakes changes |
Source: Evan Miller / standard two-proportion test conventions · 2025
Read the verdict
Significant, B higher.
Real win. Ship B, but confirm the sample is big enough that the lift is also practically meaningful, not just statistically so.
Not significant yet.
The gap could be noise. Keep running until you hit your pre set sample size, or accept there's no detectable difference.
Significant but tiny lift.
Statistically real, practically trivial. A 0.2 point lift at huge volume can be 'significant' yet not worth the complexity.
Confidence bounces above and below 95% daily.
You're peeking. Decide the sample size up front and judge once at the end, checking every day inflates false positives.
Run tests you can trust
01Set the sample size before you start
Decide how many visitors per variant you need up front, then don't judge until you're there. Calling it early is the number one way A/B tests lie.
02Don't peek and stop
Stopping the moment you cross 95% inflates false positives. Significance is only valid at the planned endpoint.
03Test one change at a time
Change the headline and the offer at once and a win tells you nothing about which mattered.
04Mind practical significance
Ask whether the lift is big enough to matter, not only whether it's statistically real.
05Run full weeks
Behaviour differs by day. Always test in whole week blocks so the weekday/weekend mix is even across variants.
06Respect the seasonality
A promo, a holiday, or a traffic source change mid test contaminates the result. Hold conditions steady.
The vocabulary
- Statistical significance
- Confidence that an observed difference is real, not the product of random chance.
- p value
- The probability of seeing a difference this large if the two variants were actually identical. Lower is stronger.
- Confidence level
- 1 minus the p value, as a percentage. 95% is the common bar.
- Control vs variant
- The original (A) versus the version you're testing against it (B).
- Relative lift
- The percentage change in conversion rate of the variant over the control.
- Two proportion z test
- The standard test for comparing two conversion rates given their sample sizes.
A/B test significance questions
Compare the two conversion rates with a two proportion z test: it produces a z score from the rates and sample sizes, which converts to a p value. If the p value is at or below your threshold (0.05 for 95% confidence), the difference is statistically significant. This tool does it as you type.
Keep going
A calculator tells you what. A call tells you what to do about it.
Send me the account behind these numbers. I'll tell you straight where the money's leaking and what I'd fix first — free, and you keep it whether you hire me or not.