On the MarketingExperiments blog, we often share tests we conduct with Research Partners. Today’s post was run on our own marketing campaign.
The team tested a promotional email for the MarketingSherpa 2012 Mobile Marketing Benchmark Report.
CONTROL
Subject Line: [Just Released] New Mobile Marketing Benchmark Report
The control featured general copy about using mobile in your 2013 marketing strategy and what tactics are working for mobile.
After evaluating the control, the team hypothesized the email did not have information about the insights prospective customers will receive from reading this benchmark report.
From that analysis, the team crafted …
Primary Research Question: Which email copy, general description or top 10 insights will have the highest clickthrough rate?
And produced this treatment …
TREATMENT #1
Subject Line: [Latest Research] 10 mobile marketing insights from brand-new research
In addition, the team wanted to understand how different segments of the audience responded …
Secondary Research Question: What is the performance difference between lists with first name and lists without first name?
From this secondary research question, the team created Treatment #2 – identical to the control, except it opens with “Hi Marketer,” instead of “Hi Full Name” – and Treatment #3, which featured the same subject line and email copy as Treatment #1 with a similar change to the email’s opening greeting.
RESULTS
What you need to understand: For this test specifically, the “Top 10” email was probably too long to work well in an email.
I asked Jon Hosier, Senior Marketing Specialist, MECLABS, for his biggest takeaway from this test.
“You can list, but don’t list everything,” Jon said.
Adam Lapp, Associate Director of Optimization and Strategy, MECLABS, agreed.
“I think it’s a friction issue,” he said. “I think when you have more than five numbers or bullets in a list it’s no longer effective. It essentially becomes a paragraph.”
It is also interesting to note while the “Top 10” copy underperformed the general copy in the aggregate, the “Top 10” copy which included full name outperformed the general copy that did not include full name.
By looking at the open rates, we can hypothesize this goes well beyond the benefits of personalization and a likely factor is the higher motivation of a segment that has shared more information with us.
If you’re not losing occasionally, you’re just treading water
Now, let’s take a look at the bigger picture. The reason I want to share this test is to give you a vote of confidence for your own testing efforts. Even we see losses, and it’s OK to see a loss.
Even more so, you should, at least occasionally, receive some losses from your testing. If you’re not seeing any losses, it means you’re not challenging the status quo enough and you’re not pushing the envelope enough. You’re shortchanging your ability to receive the really big, game-changing wins which can transform a marketing department.
Of course, take a systematic approach to your testing so you experience value from the knowledge gained from your losses. The team is cataloging this test, along with all of their tests, into a document which also houses overall lessons to help determine what works, what doesn’t and leads the team’s future efforts.
To help you understand how to gain value from your testing losses, you can watch the Negative Lifts: How a 24% loss produced a 141% increase in conversion Web clinic replay.
To inspire you for the next time you see a loss from a test, let me point you to a famous quote from Thomas Edison – “I have not failed. I’ve just found 10,000 ways that won’t work.”
Related Resources:
Email Marketing: Video email doubles clickthrough rate for Ragan Communications
Timing and Email Marketing: Sunday generated 23% higher clickthrough than Tuesday in test
Improve Your Email Programs: 5 test ideas
Email Optimization and Testing: 800,000 emails sent, 1 major lesson learned