Subject Line Test: 125% more unique clickthroughs

The results are in. A few weeks ago, we asked the MarketingExperiments blog community and the Copyblogger community for help on a subject line test. Today we’ll look at the process we went through, the results, and what you can learn from it all.

 

Background

Through a blog post on the MarketingExperiments blog and a blog post on Copyblogger, we asked for your subject line test ideas for an email that promoted Optimization Summit 2012 in Denver. We received 409 entries through the MarketingExperiments blog and 492 entries through Copyblogger.

Then came the culling.

Nathan Thompson, Senior Manager, Research and Strategy, MECLABS Conversion Group, was the lead judge for the entries on our blog, and Sonia Simone, CMO, Copyblogger Media, LLC, led her team’s effort (Sonia wrote about their process today on Copyblogger).

“Some were clever, some were text-book marketing, and others were, well … weird,” Nathan said.

“Our process for cutting down the list involved removing those that were factually incorrect first as we didn’t want to incite a riot among our readers once they opened the email and realized we had tricked them. Then we removed those that were too long, too confusing or unclear.”

Once the list was narrowed down to the acceptable subject lines, Nathan and the MECLABS Conversion Group team began grouping them into categories based on what the subject lines were attempting to convey:

• Incentive
• Curiosity
• Value exchange
• Newsworthiness
• Etc.

“Then we voted on the subject lines we felt were the best representation of each of these categories,” Nathan said.

 

The 6 Treatments

The team’s testing strategy ended up focusing on how the incentive held up compared to a more value-oriented subject line without the incentive. Here are the subject lines they selected, along with the categories they were testing for with each subject line:

1. [Optimization Summit] 3 Days to a Better Website ($300 Off Coupon Inside!) – (Value Exchange, Incentive)

2. Learn 3 tips that made 10,000 landing pages extremely successful – (Value Exchange, No Incentive)

3. Optimization Summit 2012 – Speakers List Now Up! + Save $300 Today – (Newsworthy, Incentive)

Three subject lines from the Copyblogger audience were also tested. Here are the subject lines Sonia and her team selected, along with the categories they were testing:

4. Quarterbacks aren’t the only changes being tested in Denver. – (Curiosity)

5. A scientific way to increase your conversions – (Benefits)

6. Do your landing pages pass this test? – (Fear)

Here is the body of the tested email. The body of the email stayed the same for each test treatment; the only variable that changed was the subject line.

 

 

The six treatments were then sent to subscribers of the MarketingExperiments email list as well as MarketingSherpa email lists. The lists were de-duped and split into six equal segments, and each subscriber received only one treatment.

 

Primary Research Question

The primary research question was, “Which motivation represented in the subject line will result in the highest clickthrough (Delivered/Clicks) rate?”

The KPI was unique clickthrough rate of overall emails delivered. We didn’t choose open rate as the KPI, because we wanted to avoid subject lines that were click bait but didn’t tie into the motivations of customers who could benefit from attending Optimization Summit 2012. Click bait can increase unsubscribes and decrease the result we really care about: unique clickthroughs. People don’t like to be tricked.

 

The Results

Drumroll, please …

And the winner is — “Do your landing pages pass this test?” written by Christine Parizo.

As you can see in the below screenshot from the MECLABS Test Protocol, Christine’s subject line generated a 125.9% higher unique clickthrough rate than the worst-performing subject line of the test.

 

 

 

Congratulations, Christine! You won a free ticket to Optimization Summit 2012, plus a free MarketingExperiments Landing Page Optimization Online Course.

Let’s put the magnitude of this into perspective. By simply changing a few words in one line, Christine helped drive more than twice as many visitors to our landing page. Talk about return on investment. Christine, I look forward to high-fiving you in person in Denver at Optimization Summit.

But wait, let’s take a look at the results from a different angle …

 

 

 

While Christine’s subject line was significantly different than most of the other subject lines, there was one subject line which it was not significantly different from at a 95% Level of Confidence – “Learn 3 tips that made 10,000 landing pages extremely successful.”

(Note: If you’re not familiar with statistical significant, you can find more information at Online Marketing Tests: How do you know you’re really learning anything? and Online Marketing Tests: How could you be so sure?)

That subject line was written by PaulM. We had only planned on giving one prize, but since there was no significant difference between the top two subject lines, we decided both copywriters should be rewarded. So Paul also wins a free ticket to Optimization Summit 2012 and a seat in our on-demand LPO course.

Plus … all six copywriters whose subject lines were selected to be tested will receive a 90-day free membership to Third Tribe Marketing, Copyblogger’s private community of online businesspeople.

“It should also be noted that the two top performing subject lines in clickthrough were also the top performing in terms of open rate,” Nathan said.

 

What can you learn from this test?

Of course, here’s the most important part of this blog post — how you can use this information to improve your own marketing efforts?

At a high level, the whole reason we ran this test was to show you just how powerful testing and optimization can be. As I said, the winning subject line drove more than twice as many visitors to the landing page. This isn’t some massive marketing automation or e-commerce cart redesign effort, it is simply changing a few words.

Second, test don’t guess. Some of the tops minds in marketing — like Sonia Simone and Nathan Thompson — were involved in this test, and while I didn’t put anyone on the spot, in working with them on setting up this test, I can tell you there was no clear winner in our minds.

Also, be open. Turn to those outside your normal cadre when crafting tests. That is the beauty of testing: you can always have one or two safe ones, plus one that really challenges the model. And when that outlier wins, it can open your eyes to a whole new approach.

For us, we had help from the savvy marketing professionals who read the MarketingExperiments blog and Copyblogger. While you might not have that access, you can still ask your customers for ideas through your content marketing, social media channels or surveys.

Also, you can ask an expanded peer group. Reach outside of the marketing department and ask for opinions from Sales, Customer Service, Product Development, IT, and even front-line, customer-facing employees. Beyond your company, ask peers in other companies, your husband or wife, the people you play basketball or squash with … anyone who can expand your horizons.

 

As for interpreting the results of this test itself …

The two top performers both used the specific phrase “landing pages.” The people that are interested in landing pages are interested in Optimization Summit because they understand the power of landing page optimization.

Keywords are important for more than just SEO. Google has trained us to conjure up short phrases when we want information. People think in keywords now, any time they’re looking for info, not just when they’re staring at a blank search box. When your recipients scan the subject lines in their inbox, what keywords can you provide to tap into their interests and motivation?

“The results, interesting as always, tend to indicate that incentive was not as big a player as we had guessed it would be,” Nathan said. “In fact, the two top performing subject lines didn’t mention the discount incentive at all. And even the subject line that paired the incentive with a bit of newsworthiness (Speaker’s List) was unable to outperform ‘Learn 3 tips that made 10,000 landing pages extremely successful.’”

Why wasn’t incentive effective in the subject lines? You need to sell the value of a product before anyone will find that incentive valuable. “$300 off” is not valuable, especially in an inbox cluttered with other incentive-laden subject lines. “$300 off” something your audience wants and values, now that is worthy of a clickthrough. I learned that we seriously underestimated how well our audience understands the value of Optimization Summit 2012 … or even knows of it at all.

This is a common problem many marketers make. We’ve been sleeping, eating and dreaming Optimization Summit as we have been putting it together, so it’s all too easy to overestimate how much our audience has learned about this event.

“This is a solidly specific headline – the readers know that the message will be about landing pages,” Sonia said, in analyzing the winning subject line. “Passing (or failing) tests is a common anxiety, and in a competitive economy, no one wants to have the landing pages that fail the test.”

“The word this is important (in fact, there was another entry for ‘Do your landing pages pass the test?’). That slight added element of specificity enhances reader curiosity to generate those clickthroughs.”

 

Related Resources:

April Book and Online Training Sale — Save 33% off more than 38 books and online training courses

Copywriting Contest: Write the best-performing subject line and win

Marketing Research: Top email elements to test

Email Marketing: 17.36% higher average clickthrough rate in 7 personalized subject line tests