Here’s Why Most A/B Testing is Boring and Not Really Worth the Trouble

Do a quick Google search on “things to a/b test on a website,” scan the results for a moment, then come back and read the rest of this article.

Most of you reading this are marketers, so you know I’m taking a big risk by telling you to go do something else before you read my article.

In fact if you’re reading this now, you’re probably one of the very few who made it back from that incredibly distracting activity I had you do. Thank you. You are exactly the person I want to be reading this. The others can go on their merry way. They are not the ones who need to hear this.

I had you do that search because the Internet is full of people telling you to test things on your website such as color, button size, layouts, forms, etc. I wanted you to get an idea for what’s out there.

Now, I want you to understand why almost everyone writing those articles is wrong …

… or at the very least, missing the point.

Please don’t view this as me putting down the people who wrote those articles. I know a few of them personally, and I highly respect the work they are doing. This is not about whether their work is good or bad.

I’ve personally written many articles exactly like the ones they’re writing. In fact, they have one up on me because at least their articles are ranking in Google for popular search terms.

The reason they are missing the point is that most of those articles are focused on the elements of a page rather than the serving of a customer.

I get why they do it.

Webpages are far easier to understand than people. Webpages are a collection of 0s and 1s. People are a collection of who knows what.

And most of you, readers, are looking for webpage fixes — not a deeper, fuller way to serve your customer.

There is nothing necessarily wrong with you, but it’s just that we naturally focus on our own self-interest. It isn’t wrong in itself.

What is wrong is the methods we use to achieve our own goals. I don’t mean morally wrong. I mean practically wrong.

 

Our objective should always be: Make as much money possible.

MECLABS Institute has found after more than 15 years of research that the best method for achieving this objective is to spend as much money possible on serving your customer.

Until we can view every A/B test we run as an opportunity to better serve our customers, we will just be running (ultimately) ineffective tests on page elements.

It doesn’t really matter in the long run which color, layout or page element is going to perform well.

The Internet is constantly changing. Design trends are always going to influence how we look at webpages and their elements. What matters for marketers in the long run is how well we understand and, consequently, how well we can serve our customers.

Flint McGlaughlin, Managing Director and CEO, MECLABS, calls this understanding of our customers “customer wisdom.”

This is also why he often says, “The goal of a test is not to get a lift, but rather to get a learning.”

However, it’s one thing to hear this, another to really understand what it means.

It really means we want to conduct research, not run a test.

We want to learn a tangible lesson about our customer so that we can apply it to other areas of our marketing and achieve a maximum return on the amount of time and energy we spend on testing.

Let me show you what I mean with a real-world example. Here’s what happens when you just run an A/B test that is focused on a page element. Let’s take color for instance.

You have two treatments. The only thing changed is the background color. 

 

You also have a result. In this case, the result was a 19.5% increase in clickthrough at a 92% level of confidence. But here’s where things get tricky.

 

What does this tell us about our customer?

It might tell us that …

• Better contrast is easier for customers to read
• This customer set is more attracted to darker colors
• Compared to the rest of the page, which happened to be a lighter shade of green, the darker box stood out more and attracted more clicks

However, none of those things actually obtain some truth about the customer that we can export into some other part of the site, or better, some other part of the business (product, sales, etc.).

That’s really what we’re looking for.

Imagine what would happen to your career if you were able to influence the products your company creates with a simple A/B split test.

What if you could give your CEO definitive answers to his questions about your customers? Both in a way that is much clearer and includes more data than Sales can provide.

What if you could influence not only how your company does marketing, but also how your company innovates for the future?

As management consultant and author, Peter Drucker said,

There is only one valid definition of business purpose: to create a customer. … Therefore, any business enterprise has two — and only two — basic functions: marketing and innovation.

Too often, we think of A/B tests on the Web as another marketing spend to try to increase revenue.

However, the Web is much, much more than that for any marketer who is able to grasp and execute on its possibilities.

What are those possibilities?

Well, a year or so back, MECLABS produced a tiny research brief that went mostly unnoticed by the broad public, but I think it’s still relevant today. It lays out the possibilities for using the Web as a living laboratory. So fittingly, its title is The Web as a Living Laboratory.

You can download it for free here if you’re interested in reading more.

 

You can follow Paul Cheney, Senior Manager of Partnership Content, MECLABS, on Twitter @prcheney.

 

You might also like:

Can I test more than one variable at a time? Statisticians answer some of the most commonly asked A/B testing questions

Rapidly Maximizing Conversion: How one company quickly achieved a 58.1% lift with a radical redesign

The Web as a Living Laboratory: The Three Most Important Discoveries from Over a Decade of Experimentation

Marketing Research Chart: 75% of strategic marketers use A/B testing to learn about customer behavior