It’s the same problem, every time…
Someone new to online testing, excited to get started, eager to begin crunching numbers and reporting amazing results:
- They decide on a few things that they instinctively think will raise conversions
- They make all the changes on one page, maybe two
- Setup all sorts of tracking that they’re not sure what to do with
- And push the start button thinking that their one page will see a major increase
Rarely does a test actually perform the way we expect it to, though. We believe that one set of pages will completely outperform another, when the reality is that some performance is all over the place.
When the test is complete, what will you report upstream? That the test was a wash? Or did you learn something that will bring you closer to the key insights that will have an effect on your bottom line?
Most beginners run into a test without being able to tell you what they are really testing.
They typically say, “I’m testing to see if this page increases conversions!” The problem with this statement is that it doesn’t do anything for you if the page, well, doesn’t.
In addition, if it does raise conversion, you don’t really have the slightest idea why. Instead, you get a false sense of confidence about what works on your site and what doesn’t. Then you get yourself into much deeper trouble, making assumptions for other parts of the site or customer segments that end up costing you revenue in the long term.
So how do you create tests that get you results, regardless of the test outcome?
The answer: strategic test planning. What’s strategic test planning?
It’s like following the phrase “think before you speak.” So often, we see people who are very quick to talk about something (out of their excitement) but then realize that what they’ve just said was completely wrong for that time and place. They spend most of their time making up for their mistakes instead of avoiding them altogether.
To me, strategic planning is the following:
- Taking the time to write down what you think is the problem and how exactly you’re trying to solve it
- Deciding ahead of time how you’re going to measure success and how that connects to the bottom line
- Making your test easy to understand for someone looking in from the outside
Learn from your tests to optimize all of your marketing efforts
Here are steps you can take to help each test have results you can use not just on your site, but also in your other marketing initiatives:
1. Determine the problem you’re going to solve
a. Why am I even going to test on this particular page or process?
b. Does the data implicate that there is a problem to solve (like high bounce rate)?
c. What, according to the conversion index, appears to be wrong?
2. Create a hypothesis
a. I believe visitors respond this way because of these issues…
b. These visitors are making these actions because of this content/layout
3. Decide on how you’re going to test this hypothesis
a. If the problem is the layout (or eyepath), I am going to test a new layout that appears to work better
b. If the problem is the copy, then I am going to test new copy
c. If the problem is the offer, then I’ll test this new offer
4. Decide on how you’re going to measure it
a. Am I looking for more leads overall or better quality of leads?
b. How can I measure lead quality? Do I need to connect my treatments to my CRM?
c. Am I looking for more sales or more revenue per sale? Can I connect treatment performance to my transactional database?
d. How does what I’m measuring connect to the bottom line? If I see an increase here, what does that mean for the business?
5. Decide on when you’re going to run it and how long
a. Am I testing holiday traffic or everyday traffic?
b. Do I want to see if there is a difference in behavior around certain times?
c. How long will it take me to get statistical significance (so I can prove that a change wasn’t due to random chance)?
6. Write everything down
a. For the control:
i. I believe this is the problem (hypothesis)…
ii. These are the possible solutions…
b. For each treatment:
i. This is exactly what I’m testing with this page…
ii. This is what I’m going to learn with a positive and negative result…
c. For test execution
i. This is what I’m using to measure success and why…
ii. This is how long it’s probably going to take for statistical significance…
iii. This is when I’m going to test it and why…
7. Review it with a friend or colleague?
a. Does this make sense?
b. Am I trying too hard?
In my years of testing, I’ve discovered that all optimization is relative to the visitor. Certain techniques work with some people and don’t work with others. The only way you’ll discover what really works for your visitors, for your customers, is when you spell out what you’re trying ahead of time and then see your plan through to completion.
By doing this, the treatment that “tanks” becomes the treatment that teaches you something about your audience. The more tests you run like this, the fewer chances of failure you’ll have for the future and the more opportunities you’ll have to serve your visitors and customers better.
Related Resources
Marketing Testing and Optimization: How to begin testing and drive towards triple-digit ROI gains
Online Testing and Optimization: A quick guide to LiveBall – Plus links to seven more online testing quick guides
Dances with Science: Are you better off not A/B testing?
Multivariate Testing: Can you radically improve marketing ROI by increasing variables you test?
Fundamentals of Online Testing paid training and certification course
photo by: A200Wells
Hi Jon – fantastic post! One final note I would add is to understand that part of strategic testing is not to just run one test, but to run continuous testing waves. After executing a first test, iterations of that test should be conducted immediately after a champion is determined. When there are no more fruitful gains from iterating, then it’s time to introduce a new challenger in a new test wave. Strategic testing must be part of an on-going program for long-term gains.
Thanks Kristina,
The subject of test plans, or waves, deserves its own article 🙂 There are common problems, too, that I and others have experienced in our early days of designing test plans. The first step, of course, is making sure the individual tests themselves are meaningful (hopefully this post helps!).
I’ll follow-up soon with the rest of the steps.
-Jon
Really good ideas Jon I am bookmarking and tweeting
John,
When you guys work on optimizing for a company, how many projects do you work on at one time? How long does executing this project take you?
Hi Daniel,
It largely depends on the amount of resources available to dedicate to the research…with site traffic generally being one of those resources as well as design and development resources required to execute.
Another factor is the amount of data available to-date on customer behavior (in metrics and the backend). Typically the less available, the more time is needed to collect it. Prior to running any tests, i’ve discovered that there is a minimal amount of research that needs to be done so you can design your tests to reveal more about your customer theory.
See my article regarding some of that research: https://www.marketingexperiments.com/blog/analytics-testing/segmenting-value-propositions.html
Hope this helps!
-Jon