Validity Threats: How we could have missed a 31% increase in conversions


Some marketers simply drop two landing pages into a split test tool, click a button, and then push live the page “winner” with the larger results number.

If you really want to benefit from split testing, you need to do a little more. You need a basic understanding of what you’re really doing when you’re testing. That includes validity, which is, at its most basic level, an assurance that the results from your tests actually reflect what is going on in the real world.

Let me show you an experiment with a MECLABS Research Partner in which an understanding of validity helped the team find a conversion lift that would have otherwise been missed.


Background: Consumer company that offers online brokerage services

Goal: To increase the volume of accounts created online

Primary research question: Which page design will generate the highest rate of conversion?

Test Design: A/B/C/D multi-factor split test



Click to enlarge

The control had heavily competing imagery and messages, as well as multiple calls-to-action. (Please Note: All creative samples have been anoymized.)



Click to enlarge


Most of the elements on the page are unchanged – only one block of information has been optimized.

In that block of copy, the team added a headline and bulleted copy highlighting key value proposition points.

The “Chat with a Live Agent” call-to-action was removed.

And, a large, clear call-to-action was added.



Click to enlarge


In Treatment #2, the left column remained the same, but the team removed footer elements.

This treatment features long copy with a vertical flow. Awards and testimonials were added to the right-hand column.

A large, clear call-to-action similar to Treatment #1 is at the bottom.



Click to enlarge


Treatment #3 is similar to Treatment #2, except the left-hand column width was reduced even further. In this treatment, the left-hand column has a more navigational role.

However, similar to Treatment #2, there is still a long copy, vertical flow, single call-to-action design.



Click to enlarge


According to the testing platform the MECLABS team was using, the aggregate results came up inconclusive. None of the treatments outperformed the control with any significant difference at a 95% level of confidence.


Experiment validity threat

However, the team noticed an interesting performance shift in the control and treatments toward the end of the test.

They discovered an email sent during the test skewed the sampling distribution.


Click to enlarge


 What they were able to learn from this test

After excluding the data collected after the email had been sent out, each of the treatments substantially outperformed the control with conclusive validity.


Click to enlarge


You can’t always overcome a validity threat in this manner. But, by having a better understanding of validity, you can determine whether the results you’re seeing are likely giving you intelligence about the customer that you can use to help improve your marketing efforts.

You can learn more about validity by using this free resource – Bad Data: The 3 validity threats that make your tests look conclusive (when they are deeply flawed).


Related Resources:

Webinar Testing: Slight title change produces 45% increase in clickthrough rate

Online Marketing Tests: A data analyst’s view of balancing risk and reward

Customer Theory: How we learned from a previous test to drive a 40% increase in CTR

Social Media Microsite Test: 3 lessons based on a 154% increase in leads

You might also like

Leave A Reply

Your email address will not be published.