An MEC reader wrote in response to The Power of Small Changes:
I enjoy reading your newsletter. I’m new to A/B testing (and the math too)…what confidence interval do you accept as reasonable?
I wanted to point out that Test #1 is not statistically significant to a 75% confidence interval. Furthermore, to get to a 90% CI…you’d need to run at least 30K+ people thru the test. This ‘small change’ could not be relied on.
In some of our published research, we include the results of “micro-tests,” which are exactly that: small tests that give an indication of which page in an A/B split test will perform better over the long run. We’re not trying to prove something in an academic forum, we’re simply trying to find out what works. It’s easy to get lost in formulas or statistical analysis when close observation will better accomplish your goal: to find out which of two pages (or two page elements) will create a higher conversion rate.
You bring up a good point about significance though. When implementing the improvements discovered in a micro-test, we always check to make sure overall site conversion improves. We don’t have a “magic number” we try to achieve each time for statistical significance and confidence. It varies from business to business and by testing parameters. If there is any doubt, we always verify the results in subsequent testing.
In the case you mention, the small text change did go on to create long-term results comparable to that of the micro-test.
We appreciate the feedback, as always. Best of luck in your efforts.
