If you’ve ever had a broken leg, you know it can be a painful and inconvenient experience. You can’t walk, and you are forced to rely on the people around you to complete the simplest of tasks. Who knew putting on shorts could be so difficult?
Here’s the thing … if you go the emergency room with a broken leg, you certainly don’t want to wake up from the ER with a splint on your broken toe (that you’ve been meaning to get checked out) instead of a cast on your leg. It’s great that the toe has been treated and will heal, but go ahead and try to stand up … I dare you. The toe did kind of hurt, but you could at least walk on it.
In other words, as we assess the ailments of our websites, it’s important to optimize and test elements with the greatest impact on conversion prior to treating the minor ailments. You wouldn’t want to optimize the headline of a landing page that has no button to proceed, would you?
This sounds simple, right? Well, let me tell you that it is not. Conversion rate scientists could be testing for years and not realize this one critical point.
Let’s set up a test
When you plan for a test, the first thing you want to do is analyze the page for problems, and then formulate hypotheses for fixing those problems. Here’s a hypothetical test set-up I created to help you understand this important lesson.
Example Control Page:
So what we could do is create four separate treatments to compare against the control. The assumption would be that we can gain four key learnings from this test and maximize our knowledge about the customer. Below are the example treatments along with the hypothesis for each.
Hypothesis 1 – Making it more clear what devices support Hulu (thus potential customers are sure they are actually able to use it) will increase the conversion rate.
Hypothesis 2 – Using a headline that communicates the number and diversity of titles will increase the conversion rate.
Hypothesis 3 – Stating that the site is secure will reduce anxiety of moving forward, and thus will increase conversion rate.
Hypothesis 4 – Adding bullet points that communicate the value of Hulu Plus will increase the conversion rate.
Is this an effective test set-up?
On the surface, you may think, “Yes, each treatment has only one difference from the control, so we can precisely measure its impact.” If Treatment 1 wins, we learn that customers prefer knowing that they have a device Hulu works on before signing up for the service. And, if Treatment 3 performs equal or worse than the control, then we learn that customers are not concerned with security.
However, we are operating under a FALSE assumption that we can gain four key learnings from this test and maximize our knowledge about the customer.
Why is this so? Because prospective customers may have such an immense need to know that they own a device compatible with Hulu, the other changes on the page outside of this make no difference. Bandaging my toe doesn’t help me walk any further if I still have a broken leg!
Indeed, security may have a positive impact on conversion, but it may not be fully realized unless we address the larger issue of knowing Hulu works on all devices. In the same way, optimizing the headline of a page with no button will not increase clickthrough rate, but that doesn’t mean the new headline isn’t better.
Rewind the clock …
What we should have done is determine which of the four hypotheses has the greatest impact on the customer’s decision-making process. In this hypothetical example, let’s say it is Hypothesis 1: Making it more clear what devices support Hulu (thus thus potential customers are sure they are actually able to use it) will increase the conversion rate.
If so, we should have designed a test with three to four treatments with varying approaches to prove this hypothesis true.
- One may have been the above Treatment 1.
- Another could have been simply taking the “supported devices” link out of the footer and making it more prominent.
- Another could have been including a sentence in the main copy that “Hulu works on more than 200 devices, including Apple TV, Wii, Blu-ray players, and more.”
Exhausting a single hypothesis prior to shifting to another allows us to build on top of current customer knowledge rather than simply replace it with new knowledge. A true optimization program should not seek huge wins on a single test, but rather should design a sequence of tests with each test compounding customer learnings.
With the latter, you will eventually know your customer so well that it will be virtually impossible not to have a page converting at peak performance.
Related Resources:
Landing Page Optimization: 4 test ideas for a free-trial, lead gen form page
Landing Page Optimization: How The New York Times generated a 1,052% cumulative conversion gain
Online Marketing Tests: A data analyst’s view of balancing risk and reward
Would this test also be helped with multivariate testing as well? For example, if you had 2 equally important assumptions, like Hypotheses 1 and 2, you could create the following four treatments:
1. Same as Treatment 1
2. Treatment 1 with 2’s Header
3. Treatment 2
4. Treatment 1 without “Devices”
This test could reasonably assess the validity of both Hypotheses given a reasonable sample size. It would also drive more understanding for future tests by narrowing down which Hypothesis carries the most weight in conversion.
I found this information to be quite valuable for creating split testing pages.
Conversions may not be linked to only one variable and you emphasize this information.
You have made me think about creating multiple pages for my next testing.
are you saying that one should only test the effectiveness of one message at a time due to synergy?
Hi Jeff. Great point. I actually discuss this in my previous blog post in the section about “accessorizing” your tests: https://www.marketingexperiments.com/blog/research-topics/landing-page-optimization-research-topics/fashion-optimization-lessons.html.
Your suggested number “2” compounds learnings rather than rule ideas out that may have worked if we fixed the bigger problem.
Hi Carl,
Testing one message or variable or hypothesis isn’t the point I was trying to make. I was suggesting that if you don’t attack the biggest problem on your page, you may see little to no improvement on your minor problems. Visitors may be so impacted by the bigger problem that other changes don’t matter…even though the other changes may be effective.
@Adam Lapp
understood – but the way you outlined to actually determine the ‘main problem’ is qualifying one message at a time.