A/B Testing: How to improve already effective marketing (and win a ticket to Email Summit in Vegas)


Editor’s Note: This subject line contest is no longer accepting entries. Check out “The Writer’s Dilemma:How to know which marketing copy will really be most effective” to see which entry won, why it won and what you can learn from that to further improve your own marketing.

This blog post ends with an opportunity for you to win a stay at the ARIA Resort & Casino in Las Vegas and a ticket to Email Summit, but it begins with an essential question for marketers:

How can you improve already successful marketing, advertising, websites and copywriting?

Today’s MarketingExperiments blog post is going to be unique. Not only are we going to teach you how to address this challenge, we’re going to also offer an example to help drive home the lesson. We’re going to cover a lot of ground today, so let’s dive in.


Give the people what they want …

Some copy and design is so bad, the fixes are obvious. Maybe you shouldn’t insult the customer in the headline. Maybe you should update the website that still uses a dot matrix font.

But when you’re already doing well, how can you continue to improve?

I don’t have the answer for you, but I’ll tell you who does — your customers.

There are many tricks, gimmicks and types of technology you can use in marketing, but when you strip away all the hype and rhetoric, successful marketing is pretty straightforward — clearly communicate the value your offer provides to people who will pay you for that value.

Easier said than done, of course.

How do you determine what customers want and the best way to deliver it to them?

Well, there are many ways to learn from customers, such as focus groups, surveys and social listening.

While there is value in asking people what they want, there is also a major challenge in it.

According to research from Dr. Noah J. Goldstein, Associate Professor of Management and Organizations, UCLA Anderson School of Management, “People’s ability to understand the factors that affect their behavior is surprisingly poor.”

Or, as Malcom Gladwell more glibly puts it when referring to coffee choices, “The mind knows not what the tongue wants.”

This is not to say that opinion-based customer preference research is bad. It can be helpful. However, it should be the beginning of your quest, not the end.


… by seeing what they actually do

You can use what you learn from opinion-based research to create a hypothesis about what customers want, and then run an experiment to see how they actually behave in real-world customer interactions with your product, marketing messages and website.

The technique that powers this kind of research is often known as A/B testing, split testing, landing page optimization or website optimization. If you are testing more than one thing at a time, it may also be referred to as multivariate testing.

To offer a simple example, you might assume that customers buy your product because it tastes great and because it’s less filling. Keeping these two assumptions in mind, you could create two landing pages — one with a headline that promotes that taste (treatment A) and another that mentions the low carbs (treatment B). You then send half the traffic that visits that URL to each version and see which performs better.

Here is a simple visual that Joey Taravella, Content Writer, MECLABS created to illustrate this concept: 


That’s just one test. To really learn about your customers, you must continue the process and create a testing-optimization cycle in your organization — continue to run A/B tests, record the findings, learn from them, create more hypotheses and test again based on these hypotheses.

This is true marketing experimentation, and it helps you build your theory of the customer.


Try your hand at A/B testing for a chance to win

Now that you have a basic understanding of marketing experimentation (there is also more information in the “You might also like” section of this blog post that you may find helpful), let’s engage in a real example to help drive home these lessons in a way you can apply to your own marketing challenges.

To help you take your marketing to the next level, The Moz Blog and MarketingExperiments Blog have joined forces to run a unique marketing experimentation contest.

In this blog post, we’re presenting you with a real challenge from a real organization and asking you to write a subject line that we’ll test with real customers. It’s simple; just leave your subject line as a comment in this blog post.

We’re going to pick three subject lines from The Moz Blog and three from the MarketingExperiments Blog and run a test with this organization’s customers.

Whoever writes the best performing subject line will win a stay at the ARIA Resort in Las Vegas as well as a two-day ticket to MarketingSherpa Email Summit 2015 to help them gain lessons to further improve their marketing.

Sound good? OK, let’s dive in and tell you about your client:


Subject Line Contest — Craft the best-performing subject line to win the prize

Every year at Email Summit, we run a live A/B test where the audience helps craft the experiment. We then run, validate, close the experiment and share the results during Summit as a way to teach about marketing experimentation.

We have typically run the experiment using MarketingSherpa as the “client” website to test (MarketingExperiments and MarketingSherpa are sister publications, both owned by MECLABS Institute).

However, this year we wanted to try something different, so we interviewed three national nonprofits to find a new “client” for our tests.

We chose VolunteerMatch — a nonprofit organization that uses the power of technology to make it easier for good people and good causes to connect. One of the key reasons we chose VolunteerMatch is because it is an already successful organization looking to further improve. Here is a case study explaining one of its successful implementations: Lead Management: How a B2B SaaS nonprofit decreased its sales cycle 99%.

Another reason we chose VolunteerMatch for this opportunity is that it has three types of customers, so the lessons from the content we create can help marketers across a wide range of sales models.

VolunteerMatch’s customers are:

  • People who want to volunteer (B2C)
  • Nonprofit organizations looking for volunteers (nonprofit)
  • Businesses looking for corporate volunteering solutions (B2B), to which it offers a Software-as-a-Service product through VolunteerMatch Solutions


VolunteerMatch design of experiments

After we took VolunteerMatch on as our Research Partner “client,” Jon Powell, Senior Executive Research and Development Manager, MECLABS, worked with Shari Tishman, Director of Engagement and Lauren Wagner, Senior Manager of Engagement, both of VolunteerMatch, to understand their challenges, look at their current assets and performance, and craft a design of experiments to determine what further knowledge about its customers would help VolunteerMatch improve performance.

That design of experiments features a series of split tests, including the live test we’re going to run at Email Summit as well as the test you have an opportunity to take part in by writing a subject line in the comments section of this blog post. Let’s take a look at that experiment:


The Challenge

VolunteerMatch wants to increase the response rate of its corporate email list (B2B) by discovering the best messaging possible. In order to determine that, MarketingExperiments wants to run an A/B split test.

One issue is that the B2B list is relatively smaller than the volunteer and cause list (B2C), making it harder to test in, gain statistical significance and determine which messaging is most effective.

So we’re going to run a messaging test to the B2C list. However, this isn’t without its challenges because most individuals on the B2C list are not likely to immediately connect with B2B corporate solutions messaging.

So the question is:

How do we create an email that is relevant (to the B2C list), without asking too much, that simultaneously helps us discover the most relevant aspect of the solutions (B2B) product (if any)?


The Approach — Here’s where you come in

This is where the Moz and MarketingExperiments community comes in to help.

We would like you to craft subject lines relevant to the B2C list that highlight the various benefits of the corporate solutions tool.

We have broken down the corporate solutions tool into three main categories of benefit for the SaaS product. In the comments section below include which category you are writing a subject line for along with what you think is an effective subject line.

The crew at Moz and MarketingExperiments will then choose the top subject line in each category to test. Below you will find the emails that will be sent as part of the test. They are identical, except for the subject lines (which you will write) and the bolded line in the third paragraph (that ties into that category of value).


Category #1: Proof, recognition, credibility 


Category #2: Better, more opportunities to choose from

Category #3: Ease-of-use 


About VolunteerMatch’s brand

Since we’re asking you to try your hand at crafting messaging for this example “client,” here is some more information about the brand to inform your messaging:

VolunteerMatch Identity 


VolunteerMatch Core Values

10 Things VolunteerMatch Believes:

  1. People want to do good
  2. Every great cause should be able to find the help it needs
  3. People want to improve their lives and communities through volunteering
  4. You can’t make a difference without making a connection
  5. In putting the power of technology to good use
  6. Businesses are serious about making a difference
  7. In building relationships based on trust and excellent service
  8. In partnering with like-minded organizations to create systems that result in even greater impact
  9. The passion of our employees drives the success of our products, services and mission
  10. In being great at what we do


And now, we test

You must leave your comment with your idea for a subject line before midnight on Tuesday, Jan. 13, 2015. The contest is open to all residents of the 50 states, the District of Columbia and Canada (excluding Quebec) 18 or older. If you want more info, here are the official rules.

When you enter your subject line in the comments section, also let us know which category you’re entering for (and if you have an idea outside these categories, let us know. We just might drop it in the test).

Next, Annette Promes, Chief Marketing Officer, and Cyrus Shepard, Head of SEO and Content, Moz, will pick the subject lines they think will perform best in each category from all the comments on the Moz blog, and the MarketingExperiments team will pick the subject lines we think will perform the best in each category from all the comments on the MarketingExperiments blog.

We’ll then give the VolunteerMatch team a chance to approve the subject lines based on their brand standards and test the subject lines. We’ll report back to you through the Moz and MarketingExperiments blogs which subject lines won and why they won to help you improve your already successful marketing.

So show us what you’ve got. Write your best subject lines in the comments section below. I look forward to seeing what you come up with.


You might also like

If you’re interested in learning more about marketing experimentation and A/B testing, you might find these links helpful:

A/B Split Testing — How to use A/B Split Testing to Increase Conversion Rates, Challenge Assumptions and Solve Problems

Don’t Fall Into the Trap of A/B Testing Minutiae

A/B Testing: Example of a good hypothesis

Multivariate Testing: Can you radically improve marketing ROI by increasing variables you test?

Marketing Campaign: Landing page optimization can help improve the return on your media spend

Landing Page Optimization Online Course

Online Testing Course

Beyond Marketing Kaizen: How a CMO gaining line of sight into the testing-optimization cycle can drive triple-digit ROI improvements

Online Marketing Tests: How could you be so sure?

A/B Testing: Split tests are meaningless without the proper sample size

Here’s a look at a previous subject line writing contest we’ve run to give you some ideas for your entry:

Are You a Copywriting Hotshot? Test Your Skills in an Email Marketing Duel

A/B Testing for Fun and Profit [Subject Line Writing Contest]

Announcing: Winners of the Email Subject Line Contest

Email Marketing: What elements of your offer get people to click? [Subject line contest winner announced]

You might also like
  1. Jim Maxwell says

    Category 1:
    Beyond Fluff: Proof that Volunteering Impacts Your Culture

    Category 2:
    Remove Guesswork and Anxiety Around Volunteering Efforts

    Category 3:
    Easily Track Employee Volunteering

  2. Lori Bower says

    Category 1:
    Sourpuss? Eager beaver? Inspire every employee with the perfect cause.

  3. Jan Kenyan says

    Category #1 – A Community in Need + Employee Voluntarism gives your company a heart.

    Category #2 – VolunteerMatch.org: Happy Employees + Grateful Causes = Socially Responsible Companies. This could be you!

    Category #3 – VolunteerMatch.org – Technology to help you make a socially responsible difference. Easy Peezy!

  4. Patrick says

    What’s your “Rule of Law”

  5. Greg Haywood says

    Category #1
    Subject Line: 3 Ways People Like You are Making a Difference in 2015

  6. Carrie says

    Category # 3
    [first], you will ♥ this APP

  7. Steve Hall says

    1. Does your volunteer work make a REAL difference? Here’s how to make certain.
    2. Skyrocket the results of your volunteer work – discover where you can make the biggest difference.
    3. An easy way to make a bigger impact with your volunteer work

  8. Scott says

    What landing page will the email go to? What metric will be used to decide the winner (open, click, conversion (form submission), etc.)?

    1. Daniel Burstein says

      Great questions. The email will go to the employee solutions overview page — http://solutions.volunteermatch.org/solutions/employee

      Clicks is the key metric that will decide the winner.

  9. EdC says

    #1- For People who want to volunteer (B2C)
    “Looking for meaningful volunteer work?”

    #2- Nonprofit organizations looking for volunteers (nonprofit)
    “Seeking reliable volunteers?”

    #3- Businesses looking for corporate volunteering solutions (B2B), to which it offers a Software-as-a-Service product through VolunteerMatch Solutions
    “Need help managing volunteers? Here’s a solution.”

  10. Jim Martin says

    If you are itching to help a cause, we’ll link you with a nonprofit that can scratch it.

  11. Camille Womack says

    Category #1

    Spread the feel-good in your organization: Give back.

  12. Hal Leeds says

    ” Once in a Lifetime or at least This Year Opportunity “

  13. Marie says

    Question: Are these messages going to past volunteers, or both volunteers and proto-volunteers? The first SL for Category 1 is geared toward those who contributed time in the past year. While you state that the B2B audience is small, I believe that knowing this distinction can make a difference in how you speak to them.

    Note: “PH”below refers to the pre-header text.

    Category #1: Proof, recognition, credibility

    SL: Thank you for your support in 2014
    PH: Let’s 10X last year – here’s how

    SL: The best team building step to kick off 2015
    PH: A team that volunteers is a team united

    SL: Be the proactive leader who started it all in 2015
    PH: Unite your company through volunteering

    Category #2: Better, more opportunities to choose from

    SL: We believe you are part of a dream team
    PH: Start a volunteer movement at your company

    SL: Everyone has a super power – what’s yours?
    PH: Be the key to a meaningful movement at your company

    SL: Everyone has a talent they can share
    PH: But you have the super power to mobilize

    Category #3: Ease-of-use

    SL: No good deed goes… untracked?
    PH: Track your team’s volunteering goals through our mobile app!

    SL: Look who’s done something good!
    PH: Our mobile app can track your team’s progress, and much more

    SL: You are now free to move about the office
    PH: With our mobile app, getting the team involved is easy

  14. Lucas Wagland says

    #1 – Making a real difference is just a click away
    #2 – Changing the world is not as hard as you think

  15. Joy says

    #1: What a difference a year can make
    #2: There’s no end to what you can do
    #3: Snap! And you’ve made a difference

  16. Mark Subaih says

    Category #1:
    Subject line: For people who understand the meaning of happiness.

    Category #2:
    Subject line: Did you know.. Changing others life make you happier, volunteer today.

  17. David says

    1: Don’t just meet your social responsibilities – Exceed them with VolunteerMatch.

    2: With VolunteerMatch there are so many ways YOU can make a difference to the world we live in

    3: We can’t change the world alone, the world needs YOU too. Get the VolunteerMatch app today

  18. Cynthia says

    Category #1 – Can your company make employee volunteering the real bottom line?
    Category #2 – You’ve got tons of perfect matches. Find them all at VolunteerMatch.org
    Category #3 – Almost as easy as swiping right. Connect to great causes with VolunteerMatch’s app

  19. Angela Ashton says

    #1. and #2. (First_name), set 2015 ablaze. We’ll supply the matches.

    #3. (First_name), our app finds your perfect match. But it’s not what you think.

  20. Lisa Lang says

    This would work for all 3 categories.

    Subject: Photo enclosed, do NOT bend.

    Then add an appropriate photo to the email.

  21. Joseph Skinkis says

    The catagory would be “Retirement”

    “Retire in Chiang Mai, Thailand on $1,000 per Month”

  22. Harold says

    Category #1: Proof, recognition, credibility
    Subject line: You can make the difference in 2015
    Category #2: Better, more opportunities to choose from
    Subject line: Volunteer Opportunities To Choose
    Category #3: Ease-of-use
    Subject line: Easy-Of-Use Employee Engagement App

  23. JJ McLaughlin says

    Category 2:
    Don’t just work. Do your best work with more opportunities.

  24. Dawn Beck says

    Alert: Make 2015 your year to make a difference!

    Works for any category, assumption is “From” will clearly show it is from VolunteerMatch.org”

  25. Tyler Holmes says

    Category #2: Better, more opportunities to choose from
    “Let us help you find your true calling”

  26. Emma Coulson says

    #1 – You pitch in, inspire others to do the same
    #2 – A plethora of possibilities, there are, with VolunteerMatch
    #3 – Get help in a snap

  27. Theo Romeo says

    Category 1 – Do volunteers make a difference?
    Category 2 – More help than you can handle
    Category 3 – Need a “help wanted” for those that want to help?

  28. Susan Anderson says

    Category #3
    Wish your company image was more Ashton Kutcher and less Justin Bieber?

    Category #2
    Looking for the perfect passion project to suit EVERYONE on your team?

    Category #1
    What do George, Ellen, and Angelina do better than your brand?

  29. Benedict Paul says

    Category 3.

    Jon—FREE employee volunteering mobile app . . .

  30. Doug Kessler says

    1) Give Back: how to capture your Return on Volunteering

    1) Employee volunteering is great for business. Do it right.

    1) Change the World: one employee volunteer at a time

    2) Great causes need your great people: but which ones?

    3) Pay It Forward: An easy way to get your people volunteering

  31. Vicki Klinowski says


    Enjoy a volunteer environment at work.

  32. Renee Dube says

    Category 2:
    Thousands of opportunities for your employees to say I DO

    Category 3:
    Best free app to track employee volunteering

  33. Laurie Tomasko says

    subject line for VolunteerMatch:

    Category 2: Better, more opportuities to choose from

    Make a difference. Choose how you impact the world.

  34. Michelle Orelup says

    Category #1 Connect to your passions
    Category #1 Connect your people and their passion
    Category #2 You can make a world of difference
    Category #3 Connecting you to the right volunteer opportunity

  35. Dana Wilson says

    Category: Make your volunteering count! Easily find the best match for your skills and talents!

  36. Stephanie says

    Category #1:
    , We Need Your Help Making A Difference
    You + Your Company = A Better 2015
    , Share Your Volunteering Passion At Work

    Category #2:
    CC Your Boss: 1,000+ Ways To Make A Difference (Inside)

    Category #3:
    Volunteering Made Easy (For You & Your Company)

  37. Stephanie says

    the subject lines with commas at the beginning refer to First Name personalization

  38. Mukesh Shah says

    Category 1: For People who want to volunteer (B2C)

    “Be recognized and credible by volunteering for your passions”.

    Category 3: Businesses looking for corporate volunteering solutions (B2B), to which it offers a Software-as-a-Service product through VolunteerMatch Solutions

    “Let your staff passion make them recognized, credible and drive success with free easy app”.

  39. T.J. Lowther says

    1. Topic: Better, More Opportunities
    Volunteering shouldn’t feel involuntary. Agree?

    2. Topic: Proof
    What you do matters. We can prove it.

    3. Topic: Ease-of-use
    With volunteering, just [smart]phone it in.

  40. Emilio Gomez says

    Email subject line writing contest‏

    “What are the major challenges you are facing?”

  41. Chandler Turner says

    Fishing or Phishing? Recreation vs Identity Theft!
    Spear Phishing: Latest Danger in International Hacking.

  42. Luke Thorpe says

    Category #1: Proof, recognition, credibility
    Subject line: Follow your heart
    Category #2: Better, more opportunities to choose from
    Subject line: Follow your heart
    Category #3: Ease-of-use
    Subject line: Follow your heart

  43. Paul Kaliher says

    Category #1:
    Can Your Company Be As Good a Volunteer As You Are?

  44. Marie LeDonne says

    Category 2:

    Help big hearts meet great causes with VolunteerMatch.

  45. Christina H says

    Category 2 – Today, you could change the world!
    Category 3 – Discover your cause, right from your phone!

  46. Edilma Riano says

    Category #1: Increase your employees satisfaction without raising Salaries
    Category #2: Live like you mean it – 1000’s of volunteer opportunities available
    Category #3: We change the way to choose volunteering – We make it easy – check it out

  47. Kevin Graves says

    Although using the “First Name” seems to convert higher most of the time, I recommend split testing with vs. without.

    • [First Name], increase your personal bottom line…
    • [First Name], this is how you make a difference
    • [First Name], here’s your real recognition…
    • [First Name], real recognition starts here…
    • [First Name], increase your brands exposure by…
    • [First Name], increase your company reputation by…
    • [First Name], increase your company’s recognition by…
    • [First Name], we need your help…
    • [First Name], 70% of employees want this*…
    *Stat pulled from JA Worldwide®- The Benefits of Employee Volunteer Programs 2009 Summary Report. “70% of young workers believe employers should use volunteering as a development tool” (Sorry, couldn’t find anything more current.)

    • [First Name], does employee retention equal opportunity?
    • [First Name], a little secret to increase brand awareness
    • [First Name], strengthen your brand immediately by…
    • [First Name], increase employee retention by…
    • [First Name], secret to increasing employee retention…

    • [First Name], easy way to make a difference
    • [First Name], easily improve your brand
    • [First Name], easily increase employee engagement
    • [First Name], easily improve customer satisfaction and revenue*
    *Stat pulled from JA Worldwide®- The Benefits of Employee Volunteer Programs 2009 Summary Report. “Employees demonstrate competency gains of between 14 to 17 percent as a direct result of volunteering. Improving employee attitudes by five points results in a 1.3% point improvement in customer satisfaction. Improving customer satisfaction by 1.3% points produces a 0.5% improvement in revenue.” (Sorry, couldn’t find anything more current.)

  48. Dave Goss says

    Category 1:
    Increase Employee Engagement through Volunteering!

    Category 2:
    Thousands of Volunteer Opportunities for Employees

    Category 3:
    Use Your Mobile Device to Find and Track Volunteering Efforts

  49. Lora says

    Category 1- One person-YOU can change the world

  50. Mikayla Forkes says

    Category #2

    Option 1 : For business owners wanting to affect greater change.

    Option 2: Make a difference by helping to make countless connections.

    Option 3: Affect change by creating more connections to causes.

Leave A Reply

Your email address will not be published.