A/B Testing: How to improve already effective marketing (and win a ticket to Email Summit in Vegas)


Editor’s Note: This subject line contest is no longer accepting entries. Check out “The Writer’s Dilemma:How to know which marketing copy will really be most effective” to see which entry won, why it won and what you can learn from that to further improve your own marketing.

This blog post ends with an opportunity for you to win a stay at the ARIA Resort & Casino in Las Vegas and a ticket to Email Summit, but it begins with an essential question for marketers:

How can you improve already successful marketing, advertising, websites and copywriting?

Today’s MarketingExperiments blog post is going to be unique. Not only are we going to teach you how to address this challenge, we’re going to also offer an example to help drive home the lesson. We’re going to cover a lot of ground today, so let’s dive in.


Give the people what they want …

Some copy and design is so bad, the fixes are obvious. Maybe you shouldn’t insult the customer in the headline. Maybe you should update the website that still uses a dot matrix font.

But when you’re already doing well, how can you continue to improve?

I don’t have the answer for you, but I’ll tell you who does — your customers.

There are many tricks, gimmicks and types of technology you can use in marketing, but when you strip away all the hype and rhetoric, successful marketing is pretty straightforward — clearly communicate the value your offer provides to people who will pay you for that value.

Easier said than done, of course.

How do you determine what customers want and the best way to deliver it to them?

Well, there are many ways to learn from customers, such as focus groups, surveys and social listening.

While there is value in asking people what they want, there is also a major challenge in it.

According to research from Dr. Noah J. Goldstein, Associate Professor of Management and Organizations, UCLA Anderson School of Management, “People’s ability to understand the factors that affect their behavior is surprisingly poor.”

Or, as Malcom Gladwell more glibly puts it when referring to coffee choices, “The mind knows not what the tongue wants.”

This is not to say that opinion-based customer preference research is bad. It can be helpful. However, it should be the beginning of your quest, not the end.


… by seeing what they actually do

You can use what you learn from opinion-based research to create a hypothesis about what customers want, and then run an experiment to see how they actually behave in real-world customer interactions with your product, marketing messages and website.

The technique that powers this kind of research is often known as A/B testing, split testing, landing page optimization or website optimization. If you are testing more than one thing at a time, it may also be referred to as multivariate testing.

To offer a simple example, you might assume that customers buy your product because it tastes great and because it’s less filling. Keeping these two assumptions in mind, you could create two landing pages — one with a headline that promotes that taste (treatment A) and another that mentions the low carbs (treatment B). You then send half the traffic that visits that URL to each version and see which performs better.

Here is a simple visual that Joey Taravella, Content Writer, MECLABS created to illustrate this concept: 


That’s just one test. To really learn about your customers, you must continue the process and create a testing-optimization cycle in your organization — continue to run A/B tests, record the findings, learn from them, create more hypotheses and test again based on these hypotheses.

This is true marketing experimentation, and it helps you build your theory of the customer.


Try your hand at A/B testing for a chance to win

Now that you have a basic understanding of marketing experimentation (there is also more information in the “You might also like” section of this blog post that you may find helpful), let’s engage in a real example to help drive home these lessons in a way you can apply to your own marketing challenges.

To help you take your marketing to the next level, The Moz Blog and MarketingExperiments Blog have joined forces to run a unique marketing experimentation contest.

In this blog post, we’re presenting you with a real challenge from a real organization and asking you to write a subject line that we’ll test with real customers. It’s simple; just leave your subject line as a comment in this blog post.

We’re going to pick three subject lines from The Moz Blog and three from the MarketingExperiments Blog and run a test with this organization’s customers.

Whoever writes the best performing subject line will win a stay at the ARIA Resort in Las Vegas as well as a two-day ticket to MarketingSherpa Email Summit 2015 to help them gain lessons to further improve their marketing.

Sound good? OK, let’s dive in and tell you about your client:


Subject Line Contest — Craft the best-performing subject line to win the prize

Every year at Email Summit, we run a live A/B test where the audience helps craft the experiment. We then run, validate, close the experiment and share the results during Summit as a way to teach about marketing experimentation.

We have typically run the experiment using MarketingSherpa as the “client” website to test (MarketingExperiments and MarketingSherpa are sister publications, both owned by MECLABS Institute).

However, this year we wanted to try something different, so we interviewed three national nonprofits to find a new “client” for our tests.

We chose VolunteerMatch — a nonprofit organization that uses the power of technology to make it easier for good people and good causes to connect. One of the key reasons we chose VolunteerMatch is because it is an already successful organization looking to further improve. Here is a case study explaining one of its successful implementations: Lead Management: How a B2B SaaS nonprofit decreased its sales cycle 99%.

Another reason we chose VolunteerMatch for this opportunity is that it has three types of customers, so the lessons from the content we create can help marketers across a wide range of sales models.

VolunteerMatch’s customers are:

  • People who want to volunteer (B2C)
  • Nonprofit organizations looking for volunteers (nonprofit)
  • Businesses looking for corporate volunteering solutions (B2B), to which it offers a Software-as-a-Service product through VolunteerMatch Solutions


VolunteerMatch design of experiments

After we took VolunteerMatch on as our Research Partner “client,” Jon Powell, Senior Executive Research and Development Manager, MECLABS, worked with Shari Tishman, Director of Engagement and Lauren Wagner, Senior Manager of Engagement, both of VolunteerMatch, to understand their challenges, look at their current assets and performance, and craft a design of experiments to determine what further knowledge about its customers would help VolunteerMatch improve performance.

That design of experiments features a series of split tests, including the live test we’re going to run at Email Summit as well as the test you have an opportunity to take part in by writing a subject line in the comments section of this blog post. Let’s take a look at that experiment:


The Challenge

VolunteerMatch wants to increase the response rate of its corporate email list (B2B) by discovering the best messaging possible. In order to determine that, MarketingExperiments wants to run an A/B split test.

One issue is that the B2B list is relatively smaller than the volunteer and cause list (B2C), making it harder to test in, gain statistical significance and determine which messaging is most effective.

So we’re going to run a messaging test to the B2C list. However, this isn’t without its challenges because most individuals on the B2C list are not likely to immediately connect with B2B corporate solutions messaging.

So the question is:

How do we create an email that is relevant (to the B2C list), without asking too much, that simultaneously helps us discover the most relevant aspect of the solutions (B2B) product (if any)?


The Approach — Here’s where you come in

This is where the Moz and MarketingExperiments community comes in to help.

We would like you to craft subject lines relevant to the B2C list that highlight the various benefits of the corporate solutions tool.

We have broken down the corporate solutions tool into three main categories of benefit for the SaaS product. In the comments section below include which category you are writing a subject line for along with what you think is an effective subject line.

The crew at Moz and MarketingExperiments will then choose the top subject line in each category to test. Below you will find the emails that will be sent as part of the test. They are identical, except for the subject lines (which you will write) and the bolded line in the third paragraph (that ties into that category of value).


Category #1: Proof, recognition, credibility 


Category #2: Better, more opportunities to choose from

Category #3: Ease-of-use 


About VolunteerMatch’s brand

Since we’re asking you to try your hand at crafting messaging for this example “client,” here is some more information about the brand to inform your messaging:

VolunteerMatch Identity 


VolunteerMatch Core Values

10 Things VolunteerMatch Believes:

  1. People want to do good
  2. Every great cause should be able to find the help it needs
  3. People want to improve their lives and communities through volunteering
  4. You can’t make a difference without making a connection
  5. In putting the power of technology to good use
  6. Businesses are serious about making a difference
  7. In building relationships based on trust and excellent service
  8. In partnering with like-minded organizations to create systems that result in even greater impact
  9. The passion of our employees drives the success of our products, services and mission
  10. In being great at what we do


And now, we test

You must leave your comment with your idea for a subject line before midnight on Tuesday, Jan. 13, 2015. The contest is open to all residents of the 50 states, the District of Columbia and Canada (excluding Quebec) 18 or older. If you want more info, here are the official rules.

When you enter your subject line in the comments section, also let us know which category you’re entering for (and if you have an idea outside these categories, let us know. We just might drop it in the test).

Next, Annette Promes, Chief Marketing Officer, and Cyrus Shepard, Head of SEO and Content, Moz, will pick the subject lines they think will perform best in each category from all the comments on the Moz blog, and the MarketingExperiments team will pick the subject lines we think will perform the best in each category from all the comments on the MarketingExperiments blog.

We’ll then give the VolunteerMatch team a chance to approve the subject lines based on their brand standards and test the subject lines. We’ll report back to you through the Moz and MarketingExperiments blogs which subject lines won and why they won to help you improve your already successful marketing.

So show us what you’ve got. Write your best subject lines in the comments section below. I look forward to seeing what you come up with.


You might also like

If you’re interested in learning more about marketing experimentation and A/B testing, you might find these links helpful:

A/B Split Testing — How to use A/B Split Testing to Increase Conversion Rates, Challenge Assumptions and Solve Problems

Don’t Fall Into the Trap of A/B Testing Minutiae

A/B Testing: Example of a good hypothesis

Multivariate Testing: Can you radically improve marketing ROI by increasing variables you test?

Marketing Campaign: Landing page optimization can help improve the return on your media spend

Landing Page Optimization Online Course

Online Testing Course

Beyond Marketing Kaizen: How a CMO gaining line of sight into the testing-optimization cycle can drive triple-digit ROI improvements

Online Marketing Tests: How could you be so sure?

A/B Testing: Split tests are meaningless without the proper sample size

Here’s a look at a previous subject line writing contest we’ve run to give you some ideas for your entry:

Are You a Copywriting Hotshot? Test Your Skills in an Email Marketing Duel

A/B Testing for Fun and Profit [Subject Line Writing Contest]

Announcing: Winners of the Email Subject Line Contest

Email Marketing: What elements of your offer get people to click? [Subject line contest winner announced]

You might also like
  1. Scott says

    @Daniel Burstein

    Thanks for the additional information. Here are my entries

    Category 2
    Are your co-workers helpful?

    Category 3
    1,000s to help (and an app to help you)
    Thousands to help (and an app to help you)
    1,000s of opportunities (and an app to make it easy)
    Thousands of opportunities (and an app to make it easy)

  2. Dustin Dopps says

    Category 2

    By volunteering, you have changed lives. Help us find more volunteers like you.

  3. Nancy Harhut says

    Category 1:
    Is your company as good as you are?

    Category 2:
    They won’t know unless you tell them

    Category 3:
    New, easy way to help hundreds. Please, will you?

  4. alex says

    #1 Are you thinking what I’m thinking?

  5. caroline says

    Even though I live in Scotland so am not eligible for free tickets here’s some lines.

    “I really hate volunteering, but……………”
    “I really hate volunteering, do you?”
    “Hate volunteering? Then add your voice here”.

  6. caroline says

    Whatever you do, DON’T open this email!
    Don’t volunteer or or open this email.

  7. JacksonC says

    For any of the three categories:

    – Passionate volunteers’ opinions needed
    – We need an honest volunteer’s opinion
    – [Name], what do serious volunteers think about this?

    Interchange these words as needed: passionate/serious/dedicated/real/honest

    PS. The first line of current email would work well as a preheader.

  8. Deb Monfette says

    Category #1: Proof, recognition, credibility

    Subject Line: Shari, Volunteering I think you should know about

    [Note: Part of your identity strategy is personalization. I would definitely find multiple ways to test it. I just picked your first name. Actually this subject line could be used in all 3 categories to test the call to action in each of the the emails inside.]

  9. Harvey Cahoon says

    Subject Line Submital

    Better Results with Student iPads using LanSchool

  10. Molly O'Riordan says

    #1 You and I know it’s worth it. Do they?

    #2 You’ve got passion, we’ve got options

    #3 Love to volunteer? There’s an app for that

  11. Maeghan says

    #1 [First Name], Your HR Dept Needs to See This
    #2 Get Perks at Work for Volunteering

  12. Olivia Fraustino says

    Category 1: Proof, recognition, credibility

    SL: Will you help your coworkers give back?

    Category 2: Better, more opportunities to choose from

    SL: 90,000+ opportunities (yes, even dog cuddling)

    Category 3: Ease-of-use

    SL: Volunteering just got easier.

  13. Debra says

    Category #1:
    Want to feel good and help to?
    Want to help others? Start here…
    Feel good, help others…volunteer! it’s never been easier.

    Category #2:
    Make a difference…1000’s of opportunities.
    Volunteering…great team builder. Get you’re team involved.

    Category #3:
    No more excuses…app makes volunteering easy.
    Volunteering app? Makes volunteering a breeze.

  14. Angela says

    Category #2

    Love Volunteering? Access Thousands of Ways to Make a Difference in 2015

  15. Sarah Hersack says

    Category 3
    Entice employee engagement and encourage community service — through an app.

    Meet the app that brings out the very best in all of us.

    Category 2
    Less is not more. Just ask a homeless person. Let’s help fix that.

    Category 1
    Let’s partner in changing the world. Every day.

  16. Stuart Clark says

    Cat 1: Proof, recognition, credibility
    Does your company’s mission to “make a difference” stretch this far?

    Cat 2: Better, more opportunities to choose from
    Could opportunities like these turn your people into superheroes?

    Cat 3: Ease of use
    “Doing good is good for business” (This makes it so easy)

  17. Debra says

    Category #1:
    Want to feel good and help to?
    Want help others? Start here…
    Feel good, help others…volunteering – it’s never been easier.

    Category #2:
    Make a difference…1000’s of opportunities!
    Volunteering – great team builder. Get you’re team involved.

    Category #3:
    No more excuses…app makes volunteering easy!
    Volunteering…app makes it a breeze.
    Have fun with colleagues and help others too!
    Want to help others and have fun with colleagues too?

  18. Johanna Bergstrom says

    #1 – Will you lead and inspire your colleagues to do good deeds?
    #2 – Volunteering opportunities to suit even the busiest CEOs and working parents
    #3 – Smartphone app volunteering? This makes it so easy to do a good turn

  19. Leah Adams-Chute says

    Great idea! My submissions are below. I look forward to seeing the results.

    Category #1
    You know good people. We have good causes.

    Category #2
    Are you in good company?

    Category #3
    It’s easy. Do good, feel better.

  20. Costin says

    Category 1:
    Your colleagues will thank you

    Category 2:
    1000s of Opportunities to Volunteer
    1000s of Opportunities to Volunteer (just choose 1)

    Category 3:
    App to Volunteer ? Count me in.

  21. David K says

    #1. It’s your time to volunteer
    #2. Helping your community? There’s a charity for that
    #3. Give-on-the-go! Search numerous charities on our mobile app

    #4. Connect with your community… AND your colleagues
    (Fourth category could be about the social connection that results from volunteering–in this case with the community as well as one’s colleagues. It seems like this is a primary reason why someone would want to volunteer)

  22. lahu Gawade says

    #1 – Does your business have the best engagement tool?
    #2 – Use this engagement tool and take your business to a different orbit all together.
    #3 – 2000+ business owners downloaded VolunteerMatch App, have you?

  23. Cecylia says

    Category 1: Proof, recognition, credibility
    Let kindness be your super power today. Volunteer.
    Be an office superhero. Spread the word. Make a difference.
    Everyone can be a hero. Here is how.
    Spread the good not the flu. Make a difference with your co-workers.

    Category 2: Better, more opportunities to choose from
    1000 heros for a better world.
    You can’t make a difference without making a connection. 1000 ways to connect.
    1000 ways to keep the good going in 2015.
    One company, thousand ways to make a difference

    Category 3: Ease-of-use
    Volunteering just got easier with our mobile app
    Volunteering is just a click away
    The volunteer app your co-workers talk about.
    Help is in your hands, we just make it easy.
    Let’s connect to change the world.

  24. Winnie J says

    #1 – Help your company help others

    #2 – There’s a volunteer opportunity out there for everybody

    #3 – Make it your business to help others…with one simple app.

  25. Tim says

    Category #1

    You Volunteer! Will they?

  26. Nichole A. P. says

    Category #1: Connect – Volunteer – Make a Difference Today
    Category #1: Proof That Employee Volunteerism Matters
    Category #1: Join us in the movement today!
    Category #1: We could use a little help from our friends
    Category #1: We could use a little help from your friends
    Category #1: Make the world a better place today

    Category #2: Oodles of opportunities to volunteer – Choose today!
    Category #2: Excited to volunteer but not sure where to start?
    Category #2: Volunteerism 2015 – Uniting you and your company today

    Category #3: Volunteering has never been so easy!
    Category #3: Change the world from your phone today
    Category #3: Volunteering made easy peasy!
    Category #3: Volunteering is easy peasy with this mobile app

  27. Katy says

    #1 – What Are You Willing To Do To Make A Difference?

    #2 – Find Out What Volunteer Opportunities Are Right For You

    #3 – Volunteering On The Go, Its That Easy

  28. Cecylia says

    category 1:
    The difference you and your co-workers can make in 2015
    How you and your company can better the world in 2015

    category 2:
    Thousands of opportunities for your company to make a difference
    Thousands of opportunities to better the world in 2015

    Category 3:
    Change the world with one simple app
    Make the world a better place with this free app
    Changing the world is just one click away

  29. Ashley G. says

    Category 1:

    You Matter – Explore Easy Ways for You to Make a Difference

    Category 2:

    Want to Make a Difference in Your Community? Find Your Perfect Opportunity.

    Category 3:

    Making a Difference is Easy – Find Opportunities Anytime, Anywhere

  30. Michael Monita says

    Category 1: Proof, recognition, credibility
    Bring The App That’s Powered by Your Passion to Your Company

    The App That’s Powered by the Passion of [FirstName] [LastName]

    The App That’s Powered by Your Passion

    Category 2: Better, more opportunities to choose from
    Where Motiv(ation) Meets Opportunity – The Case for Employee Engagement With VolunteerMatch

    Share thousands of volunteer opportunities with your colleagues

    Category 3: Ease-of-use
    It’s Simple: Your Volunteer Efforts X The Power of Your Colleagues = Good for All

    Now it’s easy to measure the things that matter. Employee Engagement Solutions by VolunteerMatch.

  31. Alison says

    Category #1
    Ignite a Volunteer Revolution at Work. Start here…
    Volunteer with Your Coworkers: Here’s How

    Category #2
    Your Choice: 1,000s of Volunteer Ops for Businesses

    Category #3
    On the go, like you: VolunteerMatch app for employees
    Employee Volunteering Made Easy: Get the App

Leave A Reply

Your email address will not be published.