When it comes to testing, A/B testing typically steals the spotlight, casting its sister procedure, variance testing, in the shadows. However, according to Emily Emmer, Senior Interactive Marketing Manager, Extra Space Storage, that’s a mistake.
At MarketingSherpa MarketingExperiments Web Optimization Summit 2014, Emily presented on how her team was able to utilize variance testing to transform Extra Space Storage’s Wild West testing culture into a wildly successful testing environment.
Before the team conducted variance testing, the company’s testing environment was structured like a free-for-all. There were few, if any, set rules in place, and, according to Emily, the person with the highest title and the loudest voice typically had their test implemented. All of this changed after the Extra Space Storage team ran some variance tests.
Variance testing measures two identical Web experiences to determine a site or page’s natural variability. This procedure generally constructs the rules for subsequent A/B tests to follow.
By focusing on variance testing and translating the results from this procedure into rules for A/B testing, Extra Space Storage achieved a 45% increase in conversion rate from the previous year. Watch the below excerpt to learn the results of the team’s test, the rules they developed and Emily’s advice on when to start variance testing and how to implement it.
Run variance experiments and interpret results
Emily and her team ran five simultaneous variance tests on Extra Space Storage’s homepage, which measured:
- Experiment #1: 35% traffic per treatment (two treatments)
- Experiment #2: 18% traffic per treatment (four treatments)
- Experiment #3: 12% traffic per treatment (six treatments)
- Experiment #4: 7% traffic per treatment (six treatments)
- Experiment #5: 3% traffic per treatment (six treatments)
Taking the data from these five experiments, Emily was able to extrapolate that information into rules that Extra Space Storage could use for testing moving forward, she said. Below are the results she was able to achieve:
“So you can see between Experiment 2 and Experiment 3 — really big difference, right? So moving from like a four and five percent difference in conversion rate to like a 24 and 14 percent conversion rate — that was about the breaking point,” Emily explained.
Altogether, Emily and her team developed four different testing rules from this effort, including that all tests need to run for at least two weeks.
Advice on variance testing
Emily advised anyone who runs or plans on running A/B tests to conduct variance testing beforehand. This will give future tests a baseline for how much of your traffic you can test and still expect to gain workable results.
She recommended that companies run variance testing if:
- The site has high or low points of traffic throughout the year (i.e., if there is seasonality)
- Major changes have been made to the structure of the site
- There have been any big changes in terms of the traffic
You might also like
Reserve your seat for MarketingSherpa Summit 2016 — February 22-24, 2016 at the Bellagio Resort in Las Vegas
Testing and Optimization: Variance testing for B2C contributes to 45% lift in conversion rate [MarketingSherpa video replay]
Marketing Mashup: Top takeaways from Web Optimization Summit 2014 [MarketingSherpa webinar]
B2C Marketing: Site and search optimization leads to 15% increase in average order value [MarketingSherpa case study]
