Your A/B test might be lying,before the result even loads.
A 50/50 split that quietly lands at 52/48 isn't rounding noise — it's a flashing red light. Sample ratio mismatch means something corrupted who got bucketed where, and once that's true, the lift, the p-value, and the winner are all built on sand. Paste your variant counts and find out in one chi-square test.
Your split
Visitors actually bucketed into the control variant.
Visitors actually bucketed into the treatment variant.
The split you configured. A standard 50/50 test expects 50%.
The verdict
SRM p-value
0.0
Actual split
48.5%
A vs B → 48.5% / 51.5%
Chi-square (1 dof)
8.7
Total users
10,300
Observed vs. expected
p-value vs. the 0.01 SRM alarm line
Diagnosis · likely causes
Where this mismatch is probably coming from
Redirect / extra page load
If one variant is a redirect or a separate URL, it leaks users on the bounce. The redirected arm under-counts every time — the single most common SRM cause.
Uneven bot / internal filtering
Crawlers, monitoring, and your own team landing disproportionately in one arm skew the counts. Apply identical bot and internal-IP filters to A and B.
Broken randomization or caching
Client-side bucketing with a sticky cache, flicker, or a CDN serving one variant pins users unevenly. Move assignment to the edge or server, before render.
Allocation changed mid-flight
Reweighting traffic after the test started mixes two ratios in one dataset and manufactures SRM. If this happened, restart the experiment clean.
A p-value of 0.0031 means a split this lopsided would happen by chance roughly 1 in 321times under clean random assignment. That's not noise — it's a mechanical break. Fix the plumbing, then run a fresh experiment; you can't patch SRM out of the data you already collected.
Sensitivity · how tight is your test?
Resulting p-value
0.0
At 10,300 users, drifting 1.5 percentage points off your 50% expected share for A lands a p-value of 0.0023 — enough to flag SRM. The more traffic you collect, the smaller a skew it takes to trip the alarm — which is exactly why eyeballing the percentages fails.
Reference · reading the p-value
What each band means
The 0.01 threshold is the convention because it's strict on purpose: a false alarm costs you a few minutes to investigate, while a missed mismatch costs you a wrong decision.
Your move
A clean split is worth nothing if the test asks the wrong question.
SRM p-value 0.0031 on a 48.5% / 51.5% split (10,300 users) — sample ratio mismatch, the test is compromised.
Catching SRM saves you from a bad call — but the bigger wins come from testing the right things in the right order. Bring me your experiment roadmap and I'll show you, free, which tests are worth running, which are noise, and where your real conversion leaks actually are.
Plain English
What sample ratio mismatch actually is
Sample ratio mismatch (SRM) is when the number of users in each variant of an experiment doesn't match the split you asked for. You set a 50/50 test; the data comes back 51.5/48.5. That gap feels small, but at any real sample size it's astronomically unlikely to happen by chance — and chance is the only innocent explanation.
If the assignment was truly random and balanced, the counts should hover right around the expected split. When they don't, randomization broke somewhere: a redirect that dropped users, a bot filter that hit one arm harder, a tracking tag that fired unevenly, caching that pinned people to one variant. The cause is mechanical, and it's almost always also skewing your conversions.
That's why SRM is the first thing serious experimenters check, before they ever look at the lift. A test with SRM is invalid by default — not 'a bit noisy,' invalid. The split being off means the two groups aren't comparable, so any difference you measure could be the bug, not the treatment. This checker runs the same chi-square goodness-of-fit test the big platforms use and tells you, in one number, whether to trust the experiment or throw it out.
The formula
χ² = Σ (observed − expected)² / expected, with 1 degree of freedom · SRM if p < 0.01
5,000 in A and 5,300 in B is 10,300 users. A 50/50 split expects 5,150 each. χ² = (5,000−5,150)²/5,150 + (5,300−5,150)²/5,150 = 4.37 + 4.37 = 8.74, which gives p ≈ 0.0031. Below the 0.01 threshold, so that 'small' 150-user gap is a genuine sample ratio mismatch — the test is compromised.
How to read the SRM p-value
The p-value is the probability of seeing a split this lopsided (or worse) if assignment were truly random at your expected ratio. Low means the imbalance is real, not luck. The industry-standard alarm is p < 0.01 — strict on purpose, because a false SRM alarm costs you a few minutes while a missed one costs you a wrong decision. Treat it as a band, not a knife-edge:
| p-value | What it means | Verdict |
|---|---|---|
| p ≥ 0.05 | Split is well within random variation. | Healthy |
| 0.01 ≤ p < 0.05 | Borderline — worth a glance, not an alarm. | Watch |
| 0.001 ≤ p < 0.01 | Significant mismatch. Assignment likely broke. | SRM |
| p < 0.001 | Severe mismatch. Do not read this test. | SRM |
Rule-of-thumb reading bands — not a published dataset. The p < 0.01 alarm is the industry convention; anything under it means the split is broken, not noisy.
SRM detected. Where to actually look.
One variant is a redirect or separate URL.
Redirect tests leak users on the bounce. The redirected arm loses people who abandon during the extra hop, so it always under-counts. Switch to a server-side or single-page split, or accept that redirect tests are SRM-prone by design.
Bot or internal traffic isn't filtered identically.
If crawlers, monitoring, or your own team land disproportionately in one arm, the counts skew. Apply the exact same bot and internal-IP filters to both variants, then recompute.
Assignment happens client-side with caching or flicker.
A CDN or cache that serves a sticky variant, or a flash-of-original before JS bucketing, pins users unevenly. Move the randomization earlier (edge or server) so the bucket is decided before anything renders.
The split was changed mid-flight.
Editing traffic allocation after a test starts mixes two ratios in one dataset and manufactures SRM. Never reweight a running experiment — restart it clean instead.
Counts look fine but you're slicing a segment.
SRM can be absent overall yet present inside a segment (one browser, one geo), which signals a targeting or rendering bug in that slice. Run the check on the segment, not just the total.
How to keep SRM out of your tests
01Check SRM before the lift
Run this test the moment data arrives and every time you read the result. If SRM is present, stop — the winner isn't real until the split is fixed.
02Randomize as early as possible
Edge or server-side assignment beats client-side. The earlier the bucket is set, the fewer drop-off and caching effects can skew the ratio.
03Avoid redirect-based tests
An extra page load is an extra place to lose users from one arm only. Prefer same-URL or server-rendered variants whenever you can.
04Filter both arms identically
Bots, internal users, and QA traffic must be excluded with the same rules for A and B, or you'll manufacture a mismatch.
05Lock the split for the whole run
Decide the allocation up front and never change it mid-test. Reweighting on the fly is one of the most common self-inflicted SRM causes.
06Log assignments, not just conversions
Record every bucketing decision server-side so you can audit the ratio at the source instead of trusting a downstream analytics count.
07Use a strict threshold
p < 0.01 is the convention. It's deliberately conservative so you almost never throw out a healthy test, but you also never trust a broken one.
08Re-run, don't repair, a compromised test
You can't patch SRM out of collected data. Once you find the cause, fix the plumbing and start a fresh experiment.
The vocabulary
- Sample ratio mismatch (SRM)
- A statistically significant gap between the observed variant split and the expected split. The signal that randomization broke.
- Chi-square goodness of fit
- The test that compares observed counts to expected counts. With two variants it has 1 degree of freedom.
- Degrees of freedom
- For a two-arm SRM check, 1 (number of groups minus one). It sets the shape of the chi-square distribution used for the p-value.
- p-value
- The probability of a split this extreme under true random assignment. Below 0.01 means the imbalance is almost certainly not chance.
- Expected split
- The traffic allocation you configured (e.g. 50/50, 90/10). SRM measures how far reality drifted from it.
- Bucketing
- The act of assigning a user to a variant. Faulty bucketing — drops, caching, uneven filters — is the usual root cause of SRM.
SRM questions, straight answers
Sample ratio mismatch is when the share of users in each A/B variant differs significantly from the split you configured — a 50/50 test that lands at 52/48, for example. At real sample sizes that gap is almost impossible by chance, so it means randomization broke (a redirect, a bot filter, caching, an uneven tag). When SRM is present the test is invalid, because the two groups are no longer comparable.
Keep going
A calculator tells you what. A call tells you what to do about it.
Send me the account behind these numbers. I'll tell you straight where the money's leaking and what I'd fix first — free, and you keep it whether you hire me or not.