Marketing Optimization: You can’t find the true answer without the right question

It’s the holiday time, so let me talk about a holiday. Passover, naturally (hey, if you want to succeed in marketing, don’t follow the crowds).

In the classic seder, there are The Four Sons. For this blog post, I’m going to focus on “the one who does not know how to ask a question” because I think that’s a perfect explanation of where many marketers are right now with their testing practices. For example, according to Jeff Rice’s just released 2012 Email Marketing Benchmark Report, he found that 85% of marketers don’t even know why they’re running every test they run!

–

–

Take a look at the second field in this chart. Only 15% of marketers routinely define the question, objective, and key metric when running a test. Why bother taking the time to set up a test if you don’t know what you’re looking for in the first place?

My guess is, that like the fourth son referenced above, they simply do not know how to ask. So in today’s blog post, I want to briefly discuss how to write a research question. And in this Thursday’s MarketingSherpa webinar – Negative Lifts: Turning a 25% loss into a 141% increase in conversion – Junior Editorial Analyst Paul Cheney and I will be discussing in more depth how you can learn about your customers from tests, along with Tina Hou, the director of product marketing for webinar sponsor TRUSTe.

–

How to create a clear research question

While all of the elements in Jeff’s chart are important for running a valuable test (i.e. truly learning what really works with your customers), the research question will play the biggest role in guiding your test design.

That said, the research question is just one part of step #11 (“Test Comps”) in the MarketingExperiments Optimization process. Before our analysts begin to design tests using the MECLABS Test Protocol (in which they define the research question) and begin the iterative testing process, they do everything from determining the page objective to submitting comps for peer review (you can see our full landing page optimization process in session 7 of our paid Landing Page Optimization Online Course.)

While I obviously can’t cover the entire process in this blog post, if I can help you write a true research question, I can set you on the path to learning about your customers from your tests. This is a complex process, but if I had to simplify it into three steps, I would say…

–

1. Start with asking “what” you want to know

Clearly you’re running a test for a reason. Write that question down on a piece of paper. Go ahead, do it, I’ll wait.

Now pass that piece of paper (or email) around. Are all of the key players aligned that this is, in fact, what are you trying to learn from your tests?

For example, you may want to know “What is the best price for product X?” This is the variable you will test.

–

2. Turn this into a question of “which”

Good start. Except the only problem is, to truly answer the above question, you would have to test an infinite number of prices. And I’m guessing your time, resources, traffic, and patience are not infinite.

So to narrow your focus, you want to ask a question of “which.” Not only will this force you to think about exactly how you’re designing your test, it helps you create a testing-optimization cycle to continually learn about your customers from your tests and improve your marketing performance.

A year from now, when you’re been promoted three times for driving such impressive results, and the new hotshot your direct report’s direct report hired sees a test that asks, “Which of these three price points – $1, $2, or $3 – is best for product X?” he will know exactly what you tested. And exactly what you learned about the customer.

–

3. Add in your KPI

So now you know exactly which prices (technically speaking, which values of your variable) you need to design a test around. The next question you need to ask is – how do I pick a winner?

You certainly don’t want to write the rules after the fact. “My favorite analogy for this is throwing a rock in the forest and saying, ‘look, I hit that tree,’” said Phillip Porter, Data Analyst, MECLABS. “If you aren’t aiming for something before you start, how do you know if you hit what you aimed for?”

What KPI (key performance indicator) will help you determine which value is the winner? To reformulate our example question, you would say “Which of these three price points – $1, $2, or $3 – will generate the most revenue for product X?”

Now everyone on your team (and everyone on your team a year from now) knows exactly how you define “best.” If you don’t think through and define the question beforehand, you might just try to come up with an answer based on whatever metrics you had on hand after the test is run. For example, choosing sales instead of revenue, and picking a winner that sells more product but generates less money in your pocket.

You might also not even have the chance to redefine the rules after the test is run since, since depending on the metric, the testing platform, and your transactional data system, you might not have captured the KPI that you later determine would have been most effective to know.

–

“If you don’t know where you are going, any road will get you there.” – Lewis Carroll

–

In the end, the value of the research question is that it helps ensure all the effort and resources you invest in testing and optimization gets you to where you want to go. Or, as Phillip related, you might as well be testing through the looking glass…

Alice: Would you tell me, please, which way I ought to go from here?
The Cheshire Cat: That depends a good deal on where you want to get to
Alice: I don’t much care where.
The Cheshire Cat: Then it doesn’t much matter which way you go.
Alice: …so long as I get somewhere.
The Cheshire Cat: Oh, you’re sure to do that, if only you walk long enough.

–

Related Resources:

Research Update: The state of email marketing testing and optimization

Negative Lifts: Turning a 25% loss into a 141% increase in conversion – Thursday, December 1, 1 p.m. EST

MarketingSherpa 2012 Email Marketing Benchmark Report

Landing Page Optimization: How IBM applied homepage redesign learnings to landing page testing