Promotions can be a great tool to drive traffic to your site and increase revenue.
Intuitively, it would make sense to use a promotion to boost traffic levels when testing. After all, you’re getting the traffic needed to validate the test, right?
But, how do promos really impact your testing?
What testers don’t often consider are the validity threats that promotions can have on your results.
In today’s MarketingExperiments Blog post, we’ll review how a promotion can impact testing and provide three tips if you absolutely must test during a promotion.
What is the history effect?
Promotions are a type of history effect that can pose a threat to the validity of a test.
In MECLABS’ Online Testing course, history effect is defined as “the effect an extraneous variable associated with the passage of time can have on a test variable.”
In the case of promotions, the test variable being affected by the passage of time is visitor motivation.
You’re probably thinking, “Yes, but the promotion should have a similar effect on all treatments, so how can it have a negative impact on the test if all treatments are being affected the same?”
It’s true, the promotion should in theory affect all treatments the same way.
However, what is also happening is the promotion is changing the nature and magnitude of the visitors’ motivation.
This change in motivation could be large enough to offset the effect of the test variables.
An example of history effect
To give you an example, we recently ran into this situation at MECLABS with a Research Partner that offers an online subscription service. We were running a layout test of the Partner’s subscription form with the goal to increase the subscription rate by reducing friction within the form.
The test was launched before Easter weekend, and the Partner also ran an email promotion over Easter weekend that significantly increased traffic to our test and significantly increased the subscription conversion rate for all treatments.
So, how did this affect our test?
Well, if we had analyzed the test data without knowing about the Easter promotion, we would have concluded that the treatment had no significant impact on the subscription rate – the treatment had a 6.6% relative lift over the control with only a 62% level of confidence.
However, we were aware of the promotion, knew that it presented a validity threat, and knew that it could have a significant impact on our results.
Since we were interested in the impact our test had on the visitors with “normal” motivation levels, we removed the test days from our data analysis that were impacted by the Easter promotion.
By doing so, we concluded our treatment actually did have a significant impact on the subscription rate, and were able to advise the Partner to implement the new subscription form to take advantage of the improved performance, meaning an increase in revenue. Without the days impacted by the promotion, the treatment had a 13.5% relative lift over the control with a 95% level of confidence.
If there’s no other option
So the next time you’re running a test, be sure to consider all outside variables that could have an impact on your results. You can’t control all variables, but you should be aware of them and consider how it could affect your test and how you can minimize its impact.
If you have no choice but to test during a promotional period, here are three tips to help you get the most out your testing.
Tip #1. Analyze the data with and without the promotional days
If you know there’s an outside variable that could impact your test results, such as a promotion, you should take the time to analyze the test results with the affected days removed from the results – scrub the data to exclude the promotional days. You’ll often see a difference in test results and you may surprised by how much the results can change when comparing data with the promotional days versus data without those days.
Tip #2. Testing radical redesigns
If you must run a test at the same time you’re running a promotion, make sure you’re testing a significant change (such as a radical redesign) so the effect of the testing variables outweigh the effect of the change in motivation caused by the promotion.
Tip #3. Try testing promotions instead
There are a number of tests you can run to help optimize your promotions.
Some of these include:
- Incentive type – test discounts as a percent or cash value (15% off vs. $5.00 off)
- Incentive magnitude – test different amounts (15% vs. 20%) to discover customer threshold
A promotional window can provide a great opportunity for discovering more about the offers your customers prefer.
Related Resources:
Homepage Optimization: Radical redesign ideas for multivariable testing
Optimization: A discussion about an e-commerce company’s 500% sales increase (MarketingSherpa webinar)
What about the psychology of a promotion? Is there any data to suggest certain colors and schemes work best for promotions?
Depending on who is producing the promotion, I would wonder if a promotion would perform better if it looked “fresh and exciting”. In some cases, this could require a different brand look. Attach to this promotions testing idea, this gets all sorts of murky.
Jesse;
Love the article. I especially like tip number 3, Try Testing promotions instead. Our incentive magnitude keeps changing with different products. Need to come up with a common percentage to help out our marketers. Thanks again and will work on your tips.
Teri Green
Atlas Biomechanics