The probability B beats A,not just a p-value.
A p-value tells you whether a difference is surprising under a hypothesis nobody actually believes. This tells you the thing you came for: the probability that variant B is genuinely better than A. One number, between 0% and 100%, in the language you already think in.
Your two variations
Everyone who saw the control during the test.
How many of those control visitors converted.
Everyone who saw the variant during the test.
How many of those variant visitors converted.
The verdict
P(B beats A)
96.2%
Observed uplift
16.0%
Rate A (control)
5.0%
Rate B (variant)
5.8%
Conversion rate, head to head
z-score 1.77 · clears your 95% bar
Sensitivity · how fragile is this read?
Move B's conversions, watch the probability
| B conversions | Rate B | P(B beats A) | At 95% |
|---|---|---|---|
| 260 | 5.2% | 67.5% | Hold |
| 275 | 5.5% | 86.9% | Hold |
| 290 | 5.8% | 96.2% | Ship |
| 305 | 6.1% | 99.2% | Ship |
| 320 | 6.4% | 99.9% | Ship |
Holding A fixed, a handful of conversions either way swings the probability — and sometimes the ship/hold call. The thinner your sample, the bigger each conversion's leverage, which is exactly why a flashy early lead so often evaporates.
Read it in plain language
What this number is telling you
There’s a 96.2% probability that variant B genuinely converts better than the control, given your 10,000 visitors. B is running at 5.8% against A’s 5.0% — an observed lift of 16.0%. That clears your 95% bar: you can ship B with confidence. Note: this is the answer a p-value can’t give you. The p-value asks how surprising the gap would be if B were no better; this asks the real question — how likely B is better, full stop.
Your move
A confident number is worth nothing if you're testing the wrong thing.
96.2% chance B beats A — B 5.8% vs A 5.0% (+16.0% uplift).
Most accounts don't have a stats problem — they have a too-timid-variant problem, or a sample they keep peeking at. Bring me the test behind these numbers and I'll tell you whether it's ready to ship, what to test next that's actually worth the traffic, and where your funnel has bigger wins hiding. Free call, yours to keep either way.
Plain English
Bayesian asks the question you actually have.
When you run a test, the question in your head is simple: how likely is it that B is really better than A? A Bayesian A/B test answers that directly. It returns P(B beats A), the probability that the variant's true conversion rate is higher than the control's, given the data you collected. 92% means there's a 92% chance B is the better page.
The classic frequentist test answers a different, more contorted question: if B were no better than A, how often would I see a gap this big by pure chance? That's the p-value. It's a useful guardrail, but it's not the probability B wins, and treating it as one is the most common mistake in conversion testing. The two tools are companions, not rivals.
This calculator models each variant's conversion rate as a posterior distribution and computes the chance B's rate sits above A's. We use a fast normal approximation to the Beta posteriors, which is accurate once you have a few dozen conversions per arm. It also shows your observed uplift, so you can weigh how likely B wins against how much it wins by.
The formula
P(B beats A) = Φ( (rateB − rateA) ÷ √(SE_A² + SE_B²) )
5,000 visitors each: A converts 250 (5.0%), B converts 290 (5.8%). The standard errors are about 0.31% and 0.33%, so the gap of 0.8% is roughly 1.77 standard errors out. Φ(1.77) ≈ 96%, so there's about a 96% probability B genuinely beats A — and the observed uplift is +16%. That clears the 95% bar: ship it.
How to read the probability
P(B beats A) is a continuous gauge of confidence, not a pass/fail stamp. These bands are the rule-of-thumb thresholds most teams ship against. Pick the bar that matches the cost of being wrong, then hold to it before you peek at the result:
| P(B beats A) | Call it | What to do |
|---|---|---|
| 95–100% | Ship it | Strong evidence B wins. Roll it out. |
| 80–95% | Leaning B | Promising, not proven. Keep running to clear your bar. |
| 50–80% | Coin flip | Can't separate them yet. More data or a bolder variant. |
| 0–50% | A is winning | The control is ahead. B isn't the answer here. |
Rule of thumb — no single authority sets these. Choose your bar by the cost of being wrong.
The probability isn't moving where you want. Now what?
Probability stuck near 50%.
The data genuinely can't tell A and B apart. Either the change was too small to matter, or you haven't collected enough conversions to see a real difference. Don't ship on a coin flip — keep running or test a bolder variant.
High uplift, modest probability.
B looks much better but the sample is thin, so the estimate is shaky. A big observed lift on small numbers is exactly where false winners hide. Let it run until the probability catches up to the uplift.
Probability above 95% but tiny uplift.
You're confident B wins by a hair. Ask whether a fraction-of-a-percent lift is worth the engineering and risk of the change. Statistically real and commercially worth it are not the same verdict.
Probability swinging day to day.
You're peeking and reacting. The number stabilizes as conversions accumulate; early it's noisy. Set your sample target up front, decide your shipping bar, and read the result once — not every morning.
How to run a test you can trust
01Decide the bar first
Pick your shipping threshold (90%, 95%, 99%) before the test runs. Choosing it after you see the result is how you talk yourself into a loser.
02Count conversions, not days
A test isn't 'done' on a calendar. It's done when each arm has enough conversions for the estimate to settle — usually a few hundred per variant.
03Run full weeks
Traffic and intent swing by day of week. Always run in whole-week blocks so weekday and weekend buyers are represented in both arms.
04Weigh probability against uplift
A 99% chance of a 1% lift may matter less than an 85% chance of a 30% lift. Both numbers should inform the call, not just the probability.
05Don't stop on the first peek
The probability crosses your bar and dips back below it constantly early on. Stopping the instant it pokes above 95% inflates your false-positive rate.
06Test one change at a time
If the variant changes the headline, the button, and the layout, a win tells you nothing about which one did it. Isolate the variable you want to learn from.
07Mind the segments
An overall win can hide a loss on mobile or new visitors. Once the top-line is decided, check the result holds in the segments that matter to you.
08Re-test surprising winners
If a result defies sense, the cheapest insurance is to run it again. Real effects replicate; flukes don't. A confirmed winner is worth the extra week.
The vocabulary
- P(B beats A)
- The probability that variant B's true conversion rate is higher than control A's, given the observed data. The headline Bayesian output.
- Posterior
- Your updated belief about a variant's true conversion rate after seeing the data — a distribution, not a single number.
- Observed uplift
- The relative difference between the measured rates, (rateB − rateA) ÷ rateA. How much B beats A by, separate from how sure you are.
- p-value
- The frequentist counterpart: the chance of seeing a gap this large if B were truly no better than A. Not the probability B wins.
- Credible interval
- The Bayesian sibling of a confidence interval: a range that contains the true effect with a stated probability (e.g. 95%).
- Peeking
- Repeatedly checking a running test and stopping when it looks good. It quietly inflates the rate of false winners.
Bayesian A/B testing, straight answers
It's the chance — between 0% and 100% — that variant B's true conversion rate is higher than control A's, given the visitors and conversions you entered. A result of 92% means that if you account for the uncertainty in both rates, B comes out ahead 92% of the time. That is the number people wrongly assume a p-value gives them, and it's the one this calculator computes directly.
Keep going
A calculator tells you what. A call tells you what to do about it.
Send me the account behind these numbers. I'll tell you straight where the money's leaking and what I'd fix first — free, and you keep it whether you hire me or not.