When is an A/B Split Test valid?

In response to our teleconference clinic on headline testing yesterday we received numerous emails asking questions about how to determine the validity of test results.

(The audio for the clinic is now available here. When listening to it, refer to the clinic notes as you listen.)

Here are three of the questions emailed to us:

Was the winner higher by a statistically significant amount? i.e. 0.61% vs. 0.43?

Quick question – how much data was used for this? We’ve seen some similar tests jump around so much and even perform differently when re-tested…

What number of PPC ads/impressions do we need to get an accurate result?

There were many others asking similar questions.

In almost all cases, the answer is “it depends”. That is to say, the validity of test results depends on a variety of factors.

In part, you will find answers in our A/B Split Testing brief. You’ll find recommendations on what to include in a A/B split test protocol, in addition to some testing guidelines.

A much more detailed set of answers is a central part of our Online Testing Certification Course. The current course is filled to capacity and ongoing. We will be running the course again, starting June 15th, and there are still places available.

Here is a brief description of session two of the course:

Test Validity

This may be the most vital class in the program. Invalid tests lead to incorrect decisions. This session will give you a firm understanding of validity principles, so that you can be confident in the results of your test.

• The Six Most Dangerous Threats to Test Validity – You will learn the meaning and prevention techniques associated with terms like “histriocity effect”, “instrumentation error” “sample distortion”.

• The Key Mathematic Equations For Sample Size Validity – This session will include a concise but fascinating guest lecture from an innovative professor of economics, Dr. David Reilly. You will learn the key mathematical equations for determining test size.

• The MEC Simplified Validity Formula – After the theory, we will present a simple, easy to understand formula that works even for the less mathematically inclined. You will also be given a tool to automatically calculate valid sample size.

If you feel this is the kind of detailed information you need to conduct valid A/B Split tests for your company or organization, we encourage you to sign up for the June 15th course before all the places are filled.