Multivariable Testing

We recently released the audio recording of our clinic on this topic. You can listen to a recording of this clinic here:

MultivariateTesting

Traditional thinking tells us that accurate testing of web pages, email messages, or offline media depends on us changing just one element at a time.

After all, if you change more than one element on a website sales page, and testing shows an increase or decrease in performance, how will you know which element was responsible for the change?

If you are conducting a simple A/B split test, this principle remains true. You need to test just one change at a time.

However, multivariable (or multivariate) testing allows you to test many changes simultaneously – five, ten, or even twenty. You’ll still get accurate results, without having to increase your total sample size, and you will be able to identify the impact of each individual change.

But how does multivariate testing work? Is it reliable? How does it stack up next to A/B testing?

In this brief, working with data from a variety of sources, we will show you how multivariate testing works, and how it may be able to help you dramatically and quickly improve the performance of your site pages and email messages.

Beyond the immediate improvements in page performance, multivariable testing offers two other important benefits.

1. Accelerated Learning Curve

   A single A/B split test can teach you something important about a key website page. Now imagine how much you can learn, within the same timeframe, when you look at the performance impact of ten or twenty distinct changes.

2. Breakthrough Thinking

   When you can test only one change at a time, you are under pressure to think of a “good” change… something you think has a high likelihood of delivering improved results. This can lead to cautious thinking. However, with multivariable testing you can test as many changes as you like. This takes the pressure off and gives you enormous creative latitude, opening the door to breakthrough ideas you might otherwise never have tested.

Is multivariable testing the best choice for every company and in every circumstance? Are there times when a simple A/B split test is exactly what you need?

1. How does a multivariable test differ from an A/B split test?

An A/B test isolates a single page variable and tests two variations against each other. Here are the results of a simple A/B test:

What You Need To UNDERSTAND: The optimized page improved the conversion rate of our test site from 6.00% to 6.51%, an increase of 8.5%. The 87 additional orders were worth approximately $2600 in added revenue.

In a recent brief, we covered A/B split testing in great detail. In order to get the most out of the material below, we recommend you review that article if you are unsure of how to implement effective A/B split testing.

A multivariable test transcends the limitations of a simple A/B test in two ways:

1. You can test multiple page variables at once.
2. You can use more than two variations for each variable.

With A/B split testing, it can become tedious to isolate and optimize all of the elements on a page, one at a time. In response, a number of companies have developed testing platforms that will simultaneously test a variety of elements such as graphics, background color, headline text, body copy, “call to action,” and other page constituents. Each of these elements or variables can be tested with two or more variations. The testing software splits incoming traffic among the variations and shows each visitor only one version of the composite page.

Here are the results of a multivariate micro-test. In the results below, the variables are sorted by conversion rate. This was a rather simple test, using only two variations for each of the six variables, resulting in twelve rows of results:

What You Need To UNDERSTAND: In this multivariate test, six page elements were tested simultaneously. Background color had the largest impact on conversion rate (an increase of 65.6%), followed by call to action (an increase of 56.8%).

The above test was implemented using software provided by Vertster.

Again, this was a rather simple test, using only two variations for each of the six variables, resulting in twelve rows of results.

Multivariate testing, though, can make it practical to test with many more variables and variations of each, although by increasing the variables and values, the number of visitors required for a conclusive test increases exponentially. Section Four, below, discusses how providers of multivariate testing platforms have addressed this problem.

In a subsequent multivariate test, the headline proved to be one of the most important factors that impacted the effectiveness of the page. Here are the results from just the six headline variations:

What You Need To UNDERSTAND: When weighed against the original page, headline 1 resulted in a 20.39% increase in conversion.

You can also view a screenshot showing the actual Vertster interface which generated these results:

http://meclabs.com/multivariate_headline.gif

So with all of the above, why wouldn’t you just convert to 100% multivariable tests? Multivariable testing is not the ultimate tool, but rather one of several you should use to optimize your marketing. There are times when a simple A/B test will prove superior.

A/B split testing offers the following advantages:

1. A/B testing is quick and easy to set up. You can think of a new page headline over lunch, implement a test, and see results within a few hours or days.
2. A/B testing offers unambiguous results. If you are testing only a single page variable with two possibilities, the results are typically available quickly and the best course of action is usually quite clear.
3. If you are optimizing pages for a new website, or for a client, you may want to test the existing page against a page containing all of the “best practices” you have discovered for designing landing pages, order pages, etc. The MarketingExperiments.com library of past experiments should be a helpful guide for identifying best practices in each of these areas.

Multivariable testing should be used in different circumstances:

1. It takes some time to set up a multivariable test. You have to design and implement all of the individual page variations at once.
2. However, if you have a number of variations and combinations to test, multivariable testing can save you much time and aggravation in the long run.

2. How has multivariable testing helped companies improve their marketing?

In this section, we will look at two case studies of companies who have successfully used multivariate testing to improve their marketing.

The first company, JoAnn.com, is a website serving millions of arts and crafts enthusiasts. The company used Offermatica to set up multivariate tests in a number of site areas. After one round of testing, they registered the following improvements:

What You Need To UNDERSTAND: This company was very pleased with the results of its multivariate testing, including an overall revenue per visitor improvement of 209%.

But the most memorable lesson was this: the offer that the marketing team thought would be the least viable (“buy two sewing machines and save 10 percent”) actually generated the highest return. “People were pulling their friends together and we sold enough … machines to outperform single purchases,” said Linsly Donnelly, JoAnn.com’s chief operating officer.

KEY POINT: Even the smartest marketers are often proven wrong by testing. Intuition is no substitute for well-designed experiments.

The second case study we looked at was Monster.com, a leading employment website.

In one of Monster.com’s multivariate tests (with Offermatica), they optimized their “jobs” page by testing four page elements, each with three (or two) values. These page elements were:

1. Sales Copy (Original Copy vs. Conversational Copy vs. Bulleted Copy)
2. Headlines (Default Headline vs. New Version 1 vs. New Version 2)
3. “Savings” (Savings in Percent vs. Savings in Dollars vs. No Savings Listed)
4. Savings Calculator (Present vs. Not Present)

The default page consisted of the original copy and layout, no displayed savings amount, and no savings calculator:

http://meclabs.com/monster_jobs_default.gif

The winning page included new copy, a stronger headline, savings in dollars from buying in bulk, and a savings calculator:

http://meclabs.com/monster_jobs_winner.gif

KEY POINT: The optimized page resulted in an 11.6% increase in performance.

3. What is the Taguchi Method and when should it be used?

Consider a hypothetical multivariate test that has five variables with three values each. There are 243 possible composite pages built from these elements. It may take 100 or more conversions to each of these pages to generate the most trustworthy results. If your conversion rate averages 5%, it would take 486,000 visitors to generate accurate results under these conditions (*1).

The following table illustrates the exponential increase in number of composite pages as variables and values increase:

What You Need to UNDERSTAND: As the number of variables and values increase, the number of composite pages increases at an exponential rate that often defies simple intuitive marketing design.

The Taguchi Method uses “fractional factorial testing” to reduce the number of variations necessary to determine the variables and values with the greatest impact. It was originally implemented 50 years ago and has been used successfully to test automobile and other product manufacturing. More recently, companies have begun to apply the Taguchi Method to direct marketing and the Internet. The method dictates exact combinations of page elements that allow a marketer to determine accurate estimations of the most important variables on the page, and the best values for those variables. The length of the test cycle and the number of visitors required using the Taguchi Method is surprisingly small.

For more on the Taguchi Method, see:
http://en.wikipedia.org/wiki/Genichi_Taguchi

KEY POINT: Several multivariate service providers feature Taguchi-enabled testing, which significantly reduces the amount of traffic necessary to create meaningful test results.

4. What are some of the key insights we have learned through A/B split and multivariate testing?

1. A/B split testing can be better for basic comparative testing, while multivariate testing is better for larger scale optimization. If it is cost-effective for your business, you will have the capability for both and will choose wisely between them depending on your near-term goals. Resist the urge to create large, complex tests if you are just getting started in comparative testing.
2. Although small scale micro-tests can often suggest a superior value for a given variable, the most accurate results require adequate time for a valid sample to accumulate. While the Taguchi Method can be quite effective, results containing any ambiguity should be tested for a longer period before being implemented site-wide.
3. Seasonal trends can affect the outcome of an A/B split or multivariate test if the traffic during that season is markedly different than normal traffic. Likewise, sources of traffic can affect the outcome of testing; if an influx of traffic from an unusual source comes in during an A/B micro-test, it can skew results. This further supports longer testing periods.
4. Prepare to have your intuitive expectations proven wrong. Use your intuition and experience to design good tests, but in a well-designed test with sufficient data, the numbers do not lie.
5. Don’t stop refining your marketing after just one test. Continue to optimize over the lifetime of your site. A number of small improvements over time will result in a large improvement and a highly optimized site. See our recent brief on “The Compounding Effect” to read more about this. In addition, the expectations of your customers and the Internet community as a whole may change over time; ongoing testing is the best way to keep your finger on the pulse of your customers.
6. Regarding the number of page views or visitors needed to conduct a reliable test, this can vary widely, and depends primarily the conversion rate of the page itself. A free offer page may convert at 25% or higher, while a product sales page may convert lower than 1%.

   A good rule of thumb is that you must have AT LEAST 10 conversions for each composite page in a multivariate test. A safer figure is to generate 30-50 conversions for each combination.

Keeping the above points in mind should help you get the most out of A/B and multivariate testing. Finally, we have listed a number of vendors of multivariate testing platforms in the “Literature Review” section, below.

As our future experiments reveal what really works, we will continue to share our findings. If you have any suggestions for future topics, please let us know.

(*1) To calculate the number of visitors needed to generate reliable multivariate testing, simply multiply the number of composite pages (see table above) by the historical average number of visitors required to create one conversion and by the number of conversions to create statistically significant results (typically 50-200).

V = P * C * S

V = Total Visitors
P = Composite Pages
C = Number of Visitors to Create One Conversion (average)
S = Number of Conversions to Create Statistically Significant Results

RELATED MEC REPORTS:

• A/B Split Testing
• Conversion Rate Optimization
• Long Copy vs. Short Copy

As part of our research on this topic, we have prepared a review of the best Internet resources on this topic.

Rating System

These sites were rated for usefulness and clarity, but alas, the rating is purely subjective.

* = Decent ** = Good *** = Excellent **** = Indispensable

Articles:

• Multi Variable Testing: Web Site Testing Beyond the A/B Test **
• Landing Page Optimization Using the Taguchi Method **
• Landing Page Testing Best Practices **
• Scientific Web Site Optimization using
  A/B Split Testing, Multivariable Testing, and The Taguchi Method **
• A/B Testing for the Mathematically Disinclined **
• Split Testing Ad Copy and Landing Pages **
• How to Figure out What Price to Charge via Split Testing **
• The Power of Split Testing **
• Introduction to A/B Split Testing **
• Vertster Blog **

Software/Services:

• Vertster
• Offermatica
• Optimost
• Inceptor
• ATG
• SiteSpect
• Split Run Test
• Kefta Page Optimizer
• Conversion Booster for ASP.NET
• Free A/B Split Run Testing Tool
• Add2it Go-To Pro
• MultiTrackGenerator

Credits:

Editor — Flint McGlaughlin

Writers — Brian Alt
Nick Usborne

Contributors — Jalali Hartman
Jimmy Ellis
Scott Miller
Matthew Roche

HTML Designer — Cliff Rainer