Editor’s note: During our October 14 web clinic on overcoming testing obstacles, senior researchers Boris Grinkot and Gaby Diaz fielded several audience questions on optimization testing. We’ve distilled the best questions and answers for the latest edition of our Ask an Optimizer column.
Q: How long should we split test a variable or collect data points in order to make a valid decision?
You need to use a statistical formula called the chi-square test that takes into account the number of samples you are going to collect, the amount of variance (between the control and the best performing treatment), and the statistical confidence level you desire before making a business decision (typically 95%).
Since you won’t know the variance before you start the test, you must project it to begin with and then plug the actual numbers into the formula as you conduct the test. The number of samples will be affected by the variance. The larger the variance, the fewer samples you need to collect to reach a statistically significant conclusion.
To learn more about testing, consider taking our The Fundamentals of Online Testing course, which includes our proprietary test protocol.
Q: How many tests can you have going on at the same time without skewing your results?
You can run as many multiple tests at the same time as you want, as long as they are independent and do not impact each other.
Q: What are the benefits of multivariate testing vs. A/B testing in marketing?
Multivariate testing has become quite a buzzword, so instead of focusing on the benefits, let’s discuss some things you have to take into account for a successful test. A multivariate test will be heavily impacted by some obstacles we mentioned before – time, IT resources, availability of analytics, etc.
Mainly, you need a substantial amount of testing to reach statistically significant results. On the testing web clinic, we talked about radical redesigns as a solution to a time crunch. But if you have the time, you can run a multivariate test to reach intermediate conclusions before drilling deeper with A/B split tests.
How much time do you need? As in question one, duration is affected by how many samples you have and the number of treatments. The more treatments, the more samples you have to get and the longer the test needs to run.
For example, in a single factorial A/B split test, your traffic is split just in two ways. So if you have 100,000 visitors, 50,000 see the control and 50,000 see the treatment. But if you have three headlines, three versions of the body copy, and three versions of the call to action, that becomes a fairly large number of combinations and will split your test up many more ways so it will take that much more time to get a statistically valid result.
Once a multivariate test helps you determine which elements had the biggest influence, A/B split tests will help you optimize further. For example, if you see the headline had a big influence in a multivariate test, you could perform single factorial tests to see which headline works best.
But keep in mind, on its own a multivariate test can be fairly hard to interpret. While you may find a certain combination performs well, it can be difficult to determine which of multiple variables to attribute those gains to. It is best to use multivariate testing first, then take your best guess about which elements to drill deeper into and test further.
You’ll also want to review three past research briefs that cover these areas in more depth: Multivariable Testing, A/B Split Testing, Conversion Rate Optimization Tested.
Q: How do I test two different pages with two different channels? For example, two pages with my new customers and two pages with my returning customers?
By using cookies, you can identify which customers are new and which are returning. Since returning visitors are already highly motivated and can overcome high levels of friction, you might not even want to show them a homepage at all. Perhaps show them their latest purchases or a recommendation page suggesting related items based on what they’ve purchased in the past.
Keep in mind, as we saw with the web clinic‘s second case study, it was much harder to achieve gains from the shopping cart traffic since it was already converting so high. You likely have much more room to gain from your new visitors. On the other hand, even small gains in your highest converting channels could provide a major boost to the bottom line. Testing different channels will help you understand what works best for each of them.
Q: I’ve been running a test for over a month and haven’t seen a statistical significance. Should I keep running the test or stop?
You should stop. Month-to-month variations in your traffic, due to seasonality for example, will start to skew the results. Reassess the page and launch a new test.
Q: How do you test for customer loyalty to a website?
You have to get really creative with your data analytics. One of the obstacles is prioritization. First, identify your objective. That may seem obvious – “my priority is to increase customer loyalty.”
But you have to drill deeper to understand how to measure customer loyalty to determine your key performance indicator (KPI). For example, one measurement might be return visits to your site. Working with a research partner recently, this was an important performance indicator. So KPI would be the number of visits per month from the same visitors (unique visits divided by total visits per month).
For this example KPI, you can run a test to determine which treatment is going to get people coming back to your site. You could test by sending target emails that remind them of reasons to return. If you have a retail site, perhaps the emails highlight a sales promotion. If it’s a service site, highlight an action they need to take on the site – like completing a form.
We teach a whole separate course on email optimization. But at a high level, the goal of an email is to get them to click back to your website. You don’t need all the information in the email itself. Get them to come back. And then you can test and measure the loyalty in terms of return visits.
Q: Can you test different channels with Google Analytics?
Yes. Combined with Google Website Optimizer, you can test different channels. Google Website Optimizer will track the conversion rate and statistical significance. But if you have unique URLs and mark your page with Google Analytics, you can see how they’re working in combination with your website and see the type of visitors you’re getting.
You can also see success metrics such as the conversion rate and where people navigate after that page. You can set up tests for channels such as PPC, banner advertising, or any other channel. Also, see our recent post on how to use the new features in Google Analytics.
Have additional questions? Other things you’d like to Ask an Optimizer? Use the comments section below or post your questions to our MarketingExperiments Optimization group.
Key Performance Indicators (KPI) are useful in measuring customer loyalty but personally, I would pay more attention to the visitors time on site. Instead of focusing mainly on the percentage of new visitors compared to the total visitors in a given month, the time spent by the visitors on the site can be just as valuable if not more valuable.
Hello,
Q: How do you test for customer loyalty to a website?
I like your explanation with this question. It really takes creativity and objective should be define first before creating analytical data this include a multiple types of questions to ensure customer’s loyalty to your website. But the problem sometimes, not every online survey product handles multiple styles of questions. For this reason you need to at least consider the solution of a professional survey firm. Many suggested that Zoomerang can do the job. And it’s free. Let the readers consider this: http://snurl.com/zoomtypes
More power to you and keep on writing excellent reviews! Your blog is higly recommended.