A/B tests that don't lie: when to trust the winner
You launch two versions of your landing page, three days later one is "winning" and you promote it. A month later, conversion hasn't moved. It's not bad luck: the test told you what you wanted to hear. Here's how to tell a real winner from a mirage.
The four-visit winner
With small samples, randomness rules. If version A has 5 conversions and B has 8, the difference looks like a 60% lift… but with those numbers, running the same test again could easily flip the result. It's the equivalent of tossing a coin six times, getting four heads and concluding the coin is rigged.
For an agency the cost is double: you promote a variant that isn't actually better, and you report to your client an improvement that will never show up in their bottom line.
What "significant" really means
A result is statistically significant when it's very unlikely that the difference is due to chance. The usual standard is 95% confidence: if you repeated the experiment a hundred times, the winner would keep winning in at least ninety-five of them.
The most common trap isn't ignoring statistics — it's checking the test every day and stopping the first day it turns green. That's called peeking, and it turns an honest test into a self-fulfilling prophecy: if you wait for the day randomness favours your preferred variant, it will always end up "winning".
The three rules to avoid fooling yourself
- One variable per test. If you change the headline, the price and the button at once, you'll know which version won but not why — and you won't be able to repeat the win on the next landing page.
- Full weeks. Monday traffic doesn't behave like Saturday traffic. A test running Wednesday to Friday compares different audiences without meaning to; always run whole-week cycles.
- Decide the sample size before you start. Fix how many visits or conversions you need and don't stop early, even if the scoreboard looks "obvious". The scoreboard of a half-finished test lies with a smile.
How Wirefunnels does it
Wirefunnels has the statistics built in: it splits traffic by cookie (each visitor always sees the same version), measures conversions, not clicks, and only declares a winner when there's enough sample and real confidence — until then, it honestly tells you it doesn't know yet.
When the winner exists, you promote it with one click without cutting traffic. Here's the guide to creating your first test, or try it free.