Creating a Culture of Testing: How to defeat the tyranny of best practices

You can hear Senior Manager of Research Partnerships Andy Mott answer the question How Can You Make Your Web Site Smarter? on the replay of Omniture’s latest webinar. But in my experience with these events, there is usually an interesting back story. So I cornered Andy in his office at a vulnerable time (his beloved Gators had recently lost the SEC Championship game) and found out what he really wanted to say…

 

Q: You discussed the 2009 Omniture Online Conversion Survey on a recent Omniture webinar. What surprised you the most?

Well I won’t say this surprised me. Maybe saddened is a better word. The survey asked “How frequently is online marketing testing employed in your company?” About half of the respondents said infrequently or never.

 

Q: Wow! That is pretty shocking, especially considering that these people are already familiar with testing through Omniture or MarketingExperiments. Maybe I could understand if this were the general population of marketers. But why have testing tools in place and not test? Why do you think half of them are flying blind?

Well people know they need to test. They probably know their competitors are testing and getting results. But the idea of executing a test is such a paradigm shift in the way that they’ve always done things.

Those that are higher in an organization tend to be more experienced. And if they are more experienced, they may be locked into the advertising agency way of doing things from 30 years ago, just like the doctor who overlooks recent findings and does what worked best for him when he went to medical school.

 

Q: Change is difficult. But still, thirty years ago these same people were also wearing polyester and doing the hustle. I’m a little skeptical that they would still try to shoehorn old media principles into new media.

It’s not intentional. If something has always worked for you, why change?

But what we really have is the tyranny of best practices. I’ll give you a great example. Many marketers still believe that they must have the call to action “above the fold” on a web page. Yet testing has shown this to be an utter myth.

 

Q: And nothing disproves a best practice better than a test that shows what actually works for their specific situation.

That’s the thing. Once companies start testing and see the ROI they are absolutely hooked.

 

Q: How do you take that first step? For, say, an email marketing manager reading this, how do you create a culture of testing in an organization?

Business-level executives don’t care about optimization or testing or even online marketing really. What they care about is results. So you need to talk to them in their language.

At MarketingExperiments, we publish all of our research and it is available for free. So go to the research archive and pull some experiments so you can show example results and make the business case for testing. At this point, all you are looking for is a small budget to begin testing.

Those first tests will help you establish a beachhead that you can use to further penetrate the organization. Because once businesses see the results they can gain from testing, it can get addictive. It’s like eating chips while watching a football game, you just can’t stop.

 

Q: The challenge is to just get the ball rolling. This sounds great in theory. Do you have any real-world examples?

I have countless examples. Since we started this conversion by talking about my recent webinar with Omniture, let me tell you about a Research Partner that first got interested in testing by attending an Omniture webinar that featured Dr. Flint McGlaughlin.

Companies that test usually like to stay anonymous because they view this process as such a competitive advantage. So I’ll just say they are a very large financial institution.

So this marketing manager attended Flint’s webinar and was totally sold. He was convinced that they should begin testing. But he’s only responsible for a very small patch in this giant company. It took him six months to get the approval to begin testing, doing the things I previously mentioned.

 

Q: Six months? It’s easy to get discouraged in that time. I’m not sure how many people would see it through.

But here’s the kicker. That marketing manager and his boss are now charged with trumpeting this win across the entire organization. He is now in front of his boss’s boss’s boss presenting his test results. In fact, in a few days he will be presenting in front of the SVP committee that advises the CEO.

 

Q: Well then he must have achieved some really out-of-this-world test results. What did he get…three digits…four digits? I mean, how common is that?

They got a 38% revenue boost over what the agency was doing.

 

Q: Well, that sounds decent, but a committee of SVPs really cares about 38% in one test?

You say that because you are so used to the power of testing, so you just want to see huge numbers. Let me put this another way – by not testing they would have been leaving 38% more money on the table since the cost of testing was infinitesimal compared to their massive marketing budget.

And that’s the thing. This company has a huge marketing budget. They sponsor the Olympics. They name stadiums. They purchase a ton of media. And since they don’t have space to sell in most of these executions, they’re driving everyone to the website. So if they find they could make more 38% more money without having to increase any of these huge marketing spends, the increase in ROI is humongous. Even a one or two percent increase could make or break a quarter.

 

Q: I see. I didn’t make the connection to that old media marketing spend. But I would think it goes beyond just old media driving people to a website. Online marketing is growing by leaps and bounds. I would think companies want to make sure they are getting a return on that investment as well. According to Forrester Research, digital spending will nearly double over the next five years at the expense of traditional marketing.

Forget five years from now, even today companies spend more than $25 billion on interactive marketing – things like mobile marketing, social media, email marketing, display advertising, and search marketing. That is 12% of all advertising spending. So when enterprises, like that financial institution I discussed, learn that they can take just a tiny fraction of the spend on this growing segment and invest it in a way that ensures the effectiveness of everything else they do – with real-world, statistically valid data – they get very excited.

 

Q: And I would think, for the employees that can tell management “I know how to get the best ROI from this” – not think or have an opinion, but know with real numbers – that’s quite a smart career move.

If other people are discussing so-called “best practices” and you’re showing real results, then you become the go-to person. The one who knows how this stuff really works. Because nothing defeats the tyranny of best practices as well as the audacity of testing.

And if you’re the guy that knows the right things to do in an explosively growing field like Internet marketing, while marketing budgets on everything else are falling, you’re in a good place no matter what the economy is doing.

How did you get the ball rolling on testing in your organization? What are your biggest challenges to create a culture of testing in your organization? Share your triumphs and challenges in the comments section below or post them to our MarketingExperiments Optimization group.