Testing and Optimization: A/B tests on landing pages, email and paid search from case studies


No matter what type of digital marketing campaigns you are executing, there are elements in every channel that can be tested and optimized to improve campaign performance.

For example, email subject lines, copy, design and even the “from” field can be tested. Webpage elements ripe for testing include design, layout, copy, color, call-to-action button elements and more. With paid search you should be testing keywords on an ongoing basis to continually improve your PPC spend, but you can also test ad copy and calls-to-action.

At MarketingSherpa (sister company of MarketingExperiments), we publish case studies in our newsletters every week, and very often those case studies include a testing and optimization element. For today’s MarketingExperiments Blog post, I wanted to share three of those examples taken from previously published newsletter case studies.

I hope these tests give you some ideas on testing your own digital marketing channels.


Test #1. Webpage: Increasing lead generation on a landing page

This first test was actually a collaboration between researchers at MECLABS (the parent company of MarketingExperiments) and HubSpot and was conducted during Optimization Summit 2012. The full test was covered in the article, “A/B Testing: How a landing page test yielded a 6% increase in leads.”

A lead form landing page for HubSpot’s software with a free special report incentive for filling out the registration form was tested, with the Summit attendees providing input on what to test.

Before the Summit, the testing team came up with four hypothesis options:

Hypothesis 1 — Visitors arriving to the page are highly motivated to download the e-book based on brand recognition. Removing friction from the page will result in a higher conversion rate.

Hypothesis 2 — Communicating the urgency of the offer — that the free e-book download is a limited-time offer — will result in a higher conversion rate.

Hypothesis 3 — Adding more visual value to the page, such as charts and graphs from the e-book, will result in a higher conversion rate.

Hypothesis 4 — Incorporating pricing information to increase the perceived value of the e-book will result in a higher conversion rate.

The audience was allowed to choose which one to test and decided on Hypothesis 2.




Treatment (Hypothesis 2)


The only difference between the two versions was an emblem on the page, stating, “Limited Time Offer,” to add urgency to the incentive.

The test was executed during the two days of Summit. At a 97% confidence level, it determined that the Treatment outperformed the Control 6.8%.


Test #2. Email: Testing every element in an email send

International SOS is a B2B company providing medical and travel security risk services to international corporations, governments and NGOs. The MarketingSherpa Email Newsletter case study, “Email Marketing: 400% webinar attendance increase for B2B company through relevance and A/B testing,” covered how International SOS regularly tested multiple elements in its email sends to continually optimize its campaigns.

Nadia Karasawa, Assistant Marketing Manager, International SOS, explained that emailed webinar invites were sent every month to the same audience.  That consistency made it easy for the team to A/B test email elements and benchmark against previous results.

Here are some of the elements tested and what International SOS learned about its email audience:

  • Call-to-action — the team found out that repeating the call-to-action three times offered the best results
  • “Register Now” outperformed “Register” with 4% more conversions
  • An orange call-to-action button outperformed blue or gray by 5%
  • Describing the webinar as “express,” even though the 30-minute length was not changed, increased registration and attendance
  • Other discoveries include: four bullet points outperformed three, social sharing did not improve results and both six and seven form field registrations performed equally well

What was the value of constantly testing and optimizing so many email elements?

The results the team was able to achieve were:


The 2011 send vs. 2012 send:

  • A 72% increase in registration
  • A 23% increase in attendees

The 2012 send vs. 2013 send:

  • A 47% increase in registration
  • A 23% increase in attendees


Test #3. Paid Search: Testing PPC ad copy

Testing and Optimization: Effort across entire PPC funnel leads to 83,000% boost in membership application performance,” features a test on paid search ad copy, pitting four different versions of the ad against each other. This test was run by GS1 US, a nonprofit that issues prefixes used to create U.P.C. and supply chain barcodes.

The entire optimization program covered every element of its paid search, from ad copy to landing pages and finally the application form that served as the final conversion of the PPC campaign. For this post we are focusing on the ad copy test.

The team created four versions of the PPC ad and compared results:


Ad #1:

Need to Get UPC Barcodes?
GS1 US, the only authorized source
in the U.S. for your U.P.C. barcodes

Ad #2:

Need to Get U.P.C. Barcodes?
Buy Authentic U.P.C. Barcodes
only from GS1 US Apply Online

Ad #3:

UPC Barcodes
GS1 US, the only authorized source
in the U.S. for your U.P.C. barcodes

Ad #4:

UPC Barcodes
Buy authentic U.P.C. Barcodes
only from GS1 US Apply Online



Ad #1 won the overall test with the highest clickthrough rate, besting Ad #2 by 110%, Ad #3 150.7% and Ad #4 by 252%, all at a 99.7% confidence level for the test.

One reason GS1 US tested two sets of ad titles and body text was to find the optimal combination. However, the tests were also designed to discover which language best resonated with its target audience to use for messaging in other venues, such as landing pages.


David is a Reporter for MECLABS Institute. You can follow on Twitter at @DavidOnline.


You might also like

Online Testing: Why are you really testing? [More from the blogs]

Email Testing: How the Obama campaign generated approximately $500 million in donations from email marketing [MarketingSherpa case study]

Email Marketing: Education group utilizes A/B testing to increase open rates by 39% [MarketingSherpa case study]

Email Preheaders Tested: The surprising sensitivity of a single line of text [More from the blogs]


You might also like
1 Comment
  1. Kat Hood says

    Thank you for all the great info. I will def bookmark your page.

Leave A Reply

Your email address will not be published.