A/B Testing: Split tests are meaningless without the proper sample size


A/B testing can be extremely influential. Heck, it even convinced the leader of the free world to send some curious emails (“It doesn’t have to be this way”).

This is because numbers are powerful. Seeing one call-to-action button receive a clickthrough of 10% while another button only receives 8% is pretty convincing.

Just know this …

Numbers can be misleading, as well. And one of the most frequent mistakes I see inexperienced testers make is not having a sufficient sample size.

This problem is so counterintuitive. After all, you can clearly see that 10% is more than 8%. How can that button not be better?

A quick story from my own life to explain …


Getting to the heart of testing

So my wife has a (don’t worry, it’s minor) heart thing. And she was telling me how her heart felt. And I said …

“Don’t worry, it’s perfectly normal. I get that all the time, and I’m fine.”

Well, the punch line is, I wasn’t (don’t worry, I’m all fixed now).

And that, my friends, is random chance. That’s why you want to make sure you ask enough people (or, in the case of A/B testing, have enough visits and conversions for your split test) to defeat random chance, and to know with a pretty high level of confidence that your control and treatment are actually different.


Here are a few resources to help you with sample size sufficiency in your online testing:

A/B Testing: Working with a very small sample size is difficult, but not impossible

Marketing Optimization: How to determine the proper sample size

Online Marketing Tests: How do you know you’re really learning anything?

Determining if a Data Sample is Statistically Valid

Sample Size Calculator

Marketing Optimization: How to design split tests and multi-factorial tests

You might also like
  1. Jeremy Reeves says

    Thank you! I was just debating this with a tester who claims he was some sort of “statistics expert” on a different blog… even though he thought his test showing a 47% confidence level was valid.

    Good timing 😉

Leave A Reply

Your email address will not be published.