Someone new to online testing, excited to get started, eager to begin crunching numbers and reporting amazing results:
- They decide on a few things that they instinctively think will raise conversions
- They make all the changes on one page, maybe two
- Setup all sorts of tracking that they’re not sure what to do with
- And push the start button thinking that their one page will see a major increase
Rarely does a test actually perform the way we expect it to, though. We believe that one set of pages will completely outperform another, when the reality is that some performance is all over the place.
When the test is complete, what will you report upstream? That the test was a wash? Or did you learn something that will bring you closer to the key insights that will have an effect on your bottom line?
Most beginners run into a test without being able to tell you what they are really testing.
They typically say, “I’m testing to see if this page increases conversions!” The problem with this statement is that it doesn’t do anything for you if the page, well, doesn’t.
In addition, if it does raise conversion, you don’t really have the slightest idea why. Instead, you get a false sense of confidence about what works on your site and what doesn’t. Then you get yourself into much deeper trouble, making assumptions for other parts of the site or customer segments that end up costing you revenue in the long term.
So how do you create tests that get you results, regardless of the test outcome?
The answer: strategic test planning. What’s strategic test planning?
It’s like following the phrase “think before you speak.” So often, we see people who are very quick to talk about something (out of their excitement) but then realize that what they’ve just said was completely wrong for that time and place. They spend most of their time making up for their mistakes instead of avoiding them altogether.
To me, strategic planning is the following:
- Taking the time to write down what you think is the problem and how exactly you’re trying to solve it
- Deciding ahead of time how you’re going to measure success and how that connects to the bottom line
- Making your test easy to understand for someone looking in from the outside
Learn from your tests to optimize all of your marketing efforts
Here are steps you can take to help each test have results you can use not just on your site, but also in your other marketing initiatives:
1. Determine the problem you’re going to solve
a. Why am I even going to test on this particular page or process?
b. Does the data implicate that there is a problem to solve (like high bounce rate)?
c. What, according to the conversion index, appears to be wrong?
2. Create a hypothesis
a. I believe visitors respond this way because of these issues…
b. These visitors are making these actions because of this content/layout
3. Decide on how you’re going to test this hypothesis
a. If the problem is the layout (or eyepath), I am going to test a new layout that appears to work better
b. If the problem is the copy, then I am going to test new copy
c. If the problem is the offer, then I’ll test this new offer
4. Decide on how you’re going to measure it
a. Am I looking for more leads overall or better quality of leads?
b. How can I measure lead quality? Do I need to connect my treatments to my CRM?
c. Am I looking for more sales or more revenue per sale? Can I connect treatment performance to my transactional database?
d. How does what I’m measuring connect to the bottom line? If I see an increase here, what does that mean for the business?
5. Decide on when you’re going to run it and how long
a. Am I testing holiday traffic or everyday traffic?
b. Do I want to see if there is a difference in behavior around certain times?
c. How long will it take me to get statistical significance (so I can prove that a change wasn’t due to random chance)?
6. Write everything down
a. For the control:
i. I believe this is the problem (hypothesis)…
ii. These are the possible solutions…
b. For each treatment:
i. This is exactly what I’m testing with this page…
ii. This is what I’m going to learn with a positive and negative result…
c. For test execution
i. This is what I’m using to measure success and why…
ii. This is how long it’s probably going to take for statistical significance…
iii. This is when I’m going to test it and why…
7. Review it with a friend or colleague?
a. Does this make sense?
b. Am I trying too hard?
In my years of testing, I’ve discovered that all optimization is relative to the visitor. Certain techniques work with some people and don’t work with others. The only way you’ll discover what really works for your visitors, for your customers, is when you spell out what you’re trying ahead of time and then see your plan through to completion.
By doing this, the treatment that “tanks” becomes the treatment that teaches you something about your audience. The more tests you run like this, the fewer chances of failure you’ll have for the future and the more opportunities you’ll have to serve your visitors and customers better.
Online Testing and Optimization: A quick guide to LiveBall – Plus links to seven more online testing quick guidesA200Wells