Email Testing: Subject line increases opens 12.7% … here’s why we’ll never use it again


The purpose of a subject line is to get an open. However, the purpose of a subject line is not only to get an open. In our recent subject line contest, there were some curious submissions that made me think this blog post was necessary:

  • Mom told me to wear clean underwear in case I was in an accident. I wish she had told me about this too …
  • A priest, a rabbi and a MECLABS LPO expert walk into a bar in Denver …
  • RE: The video showed a bald man. Why?
  • 1 Thing You MUST Do in Denver Before You Die
  • Want to find the end of a rainbow AND the pot of gold?
  • Denver Flight #ME430 [Confirm your seat]
  • Open &; Enjoy Real Bacon Smell…
  • This is your brain on email.
  • What the #@*? What should you test next?

I know some of these are likely meant as jokes, but if the amount of misleading subject lines that fill my own inbox every day is any indication, many of them are probably serious.


The road to unsubscribes is paved by good intentions

And I don’t mean to pick on our contest entrants. If fact, I’m going to show you a subject line test I ran making this same mistake later in this post. Even in the blog post, “Announcing the Winner of the Email Subject Line Copywriting Contest!” by Sonia Simone, CMO, Copyblogger Media, she discusses how I pushed back on one of her selections because I was worried it was misleading …

For example, we really liked the header “Testing — does this link work for you?” (We defined that one as a Curiosity headline.) But MECLABS had some concerns that the element of trickery would annoy their subscribers and lead to unsubscribes … definitely not the result we were after.

So why do we, as marketers, come up with these misleading subject lines? Well, I think our intentions are in the correct place: We’re simply looking to stand out in a crowded marketplace. However …


Email is a circle of trust

Keep in mind that by having someone’s email address and the permission to send them email, you are entering into a circle of trust in that person’s life.

Sounds kind of touchy feely, doesn’t it? But think about it. The inbox is where people share some of the most important things in life with the people that matter most to them – from important results at work to moments with the family or friends at home.

If you mislead, you violate that trust, and your audience will fight back by unsubscribing, or, perhaps even worse, labeling you as a spammer.


… and in the end, curiosity doesn’t work very well anyway

Sonia puts these types of subject lines into the greater “Curiosity” category. And while there are ways to raise curiosity without being misleading, the curiosity subject line that we ended up going with for the recent subject line contest – Quarterbacks aren’t the only changes being tested in Denver. – was the worst performer of the six subject lines we tested.

“Curiosity headlines rarely do well. Writers love them, but readers tend not to respond. That’s not something we invented or discovered; serious copywriters have been preaching it for more than 100 years now,” Sonia said.

“We’ve seen it over and over again when one of us falls in love with a clever headline idea.”

Of course, to understand what really works, you must have your eye on the correct KPI, and as I began this blog post by saying, that may not be the open rate when you’re testing a subject line.

In a recent case study email sent to the MarketingSherpa Email Marketing Newsletter list, I fell in love with a clever subject line written by our freelance reporter, Jeri Dube (note: MarketingExperiments and MarketingSherpa are both owned by MECLABS).


Today’s test is not unrelated to this milieu …

I thought, “What a perfect chance to run a test that challenges the model.”

The MarketingSherpa subject line model is pretty straightforward – we write case studies and how-to articles that focus on “what really works” in marketing, and write straightforward subject lines that reflect the results from those articles when we distribute them.

Control: How Blockbuster Express grew its email list 300%

Treatment: If Alfred Hitchcock wrote emails, could he grow an audience by 300%?

It was the perfect crime. Two unrelated subject lines meet on a train …

Sorry, that curiosity-driven subject line really got me going. And the audience, too. It generated 12.7% more opens.

But this test has a Hitchcockian twist. Even though the subject line had a higher unique open rate, it had a lower unique clickthrough rate (measured as delivered to click).

So even though the Treatment had a head start (if more people open it, there is the possibility for more people to click), it received less unique clicks.


The Control made an honest promise: “If you open this email, you will learn more about this case study.”

The Treatment was basically just saying: “Hey, dude, wanna see something cool?”

That is why I’m sticking to the tried-and-true model for the MarketingSherpa editorial subject lines. It’s always good to challenge that model, but it’s essential to focus on your true KPI. I don’t care how many opens we get if we make a misleading proposition that discourages the marketer from reading the article.


Related Resources:

Copywriting Contest: Write the best-performing subject line and win

Subject Line Test: 125% more unique clickthroughs

Email Marketing: 17.36% higher average clickthrough rate in 7 personalized subject line tests

Using an Employee Name in the ‘From’ Line: 6 Tactics to Supplement HTML Promo Emails

You might also like
  1. Jeremy Reeves says

    This is something I’ve been preaching for a long, long time and very few people really get it. I know when I look at my own email stats, I just “glance” at the opens but don’t think about them too much.

    After all, the point of an email isn’t to get read (maybe sometimes, if it’s a credibility builder or something along that nature)… but to lead them to the page with your main call to action on it.

    Jeremy Reeves

  2. Gene Gerwin says

    Even if the treatment won, I’d be hesitant to use it for fear of getting spam complaints. Your sender score’s health should be a consideration in all email campaigns.

  3. Dannie says

    What was your click/open rate on this campaign?

    Is this a metric you guys track or find value in?

    1. Daniel Burstein says


      The control had a 12.5% clickthrough rate, and the treatment had a 8.5% clickthrough rate (when measuring CTR as opens to clicks, as opposed to delivereds to clicks which I reference in the blog post).

      As for how valuable this metric is… I’ve noticed some marketers tend to like this metric, because the number is much bigger and it makes your email marketing efforts look more successful. (these numbers are based on uniques, not total, which usually inflates the number even more)

      But, my advice, is the metric you use depends on what you’re testing. The metric should never be an ego boost (let’s call that a mirror), but rather a window into what you’re trying to learn.

      For the test in this blog post, seeing how the subject line impacted the number of the people who got through the funnel and reached a MarketingSherpa article was the KPI I was focused on.

      However, if we were testing, say, body copy in an email, then click/open rate would be a more helpful metric.

      I hope this helps answer your question, Dannie, and I look forward to your thoughts as well. Thanks.

Leave A Reply

Your email address will not be published.