What started as a little humor added to a reminder email about an upcoming split test became a learning experience on testing value proposition and discovering opportunities for optimization.
In today’s MarketingExperiments blog post, we’ll share the results of our experiment and what we learned from putting these contenders to the test.
First, let’s get a little background on the test design.
The idea for this test came from a brainstorming session between Paul Cheney, Editorial Analyst, MECLABS, and Austin McCraw, Senior Editorial Analyst, MECLABS. The goal for Paul and Austin was to discover which segments of the value proposition for MarketingExperiments Web clinics have the greatest appeal and where opportunities for optimization exist.
They used a Web clinic invite email to run their subject line test. The research question was, “Which email subject line will get the highest open rate?”
When I asked Paul why they chose a subject line test instead of a split test of the title, he explained that a subject line test allowed them to test their hypothesis and keep the Web clinic title congruent.
“Because we couldn’t exactly change the title of a live event while we were trying to promote it, we needed a subtler way of testing the hypothesis. An email subject line was perfect because it has the same effect of a headline, but we didn’t need to change the title of the clinic,” Paul said.
Below is a screenshot of the Web clinic invite used for the test. All of the copy, images and the call-to-action were kept identical for both the control and the treatment.
Control Subject Line: Promise – “How to optimize your content marketing”
Paul and Austin hypothesized that if the control outperformed the treatment, then a more “how-to” based approach would be more appealing to the MarketingExperiments audience. Based on the test future clinics would then be angled toward step-by-step teaching.
Treatment Subject Line: Intrigue – “Does content marketing really work?”
If the treatment outperformed the control, however, then it might be assumed that more intriguing philosophical questions hold more appeal for the audience. In this case, future clinics could be angled toward scientific discovery and learning.
It’s important to remember here, though, that both how-to teaching and scientific discovery are facets of the MarketingExperiments value proposition. This test is simply a matter of answering the question: Which facet is more appealing?
That kind of knowledge is invaluable when our writers are deciding on clinic topics and creating the copy that promotes them.
Results
The treatment outperformed the control by a relative difference of 17% at a level of confidence of 95%. So, what can be learned about value propositions and subject lines from this test?
Don’t speculate – plan and test
After Paul and Austin ran the test, they discovered that (at least in this case) a more discovery-oriented value proposition won out. In other words, MarketingExperiments readers valued discovery over how-to teaching when it comes to content marketing.
While this certainly bears more testing, it was a fascinating insight into our audience for the team to discover.
So, the next time your team wants to understand which elements of your value proposition have a stronger appeal with your audience … don’t leave it up to speculation, plan and test your way into better serving your customers.
The results might surprise you.
“Honestly, I was surprised,” Paul said. “I thought MarketingExperiments readers would go for a more practical approach.”
Stay tuned to the MarketingExperiments blog for follow-up tests as Paul and Austin dive deeper into crafting their customer theory of Web clinic attendees.
Related Resources:
5 Steps to Effective Content Marketing: How to extract the maximum revenue from your content (Web clinic video replay)
Webinar Email Testing: 2.7% increase in CTR from extremely simple change
Value Proposition Development: 5 insights to help you discover your value prop
Webinar Testing: Slight title change produces 45% increase in clickthrough rate
“…a more discovery-oriented value proposition won out”. I don’t think the results tell you that. The control talks about “marketing” (very generic) and the treatment talks about “content marketing” (far more specific). This could easily account for the different results.
Surely, to be confident that the result showed a difference between a how-to and a discovery-oriented subject line, the control subject line should have been “How to optimise your content marketing”.
Hi Steve,
Thank you for your comments. Initially, I was a little puzzled by your response, but upon further review the confusion rests in a typographical error in the control of which I have corrected. Your observation was correct in that the control should be “How to optimize content marketing” and in fact, that was the control we tested. My apologies for the confusion and thank you for pointing that out and holding me accountable.