Professional infiltrator should seriously be added to my job description. Because once again, I gathered some marketing “intel” from somewhere that I wasn’t quite, how do I put it…invited to?
This time, I stepped out of the MECLABS classroom and into a beautiful, oceanfront hotel in Jacksonville Beach for a MarketingExperiments Landing Page Optimization Workshop. There, I blended into a crowd of about 70 marketers; listening to the presenters, Director of Training, Chuck Coker, Senior Optimization Manager, Adam Lapp (Mr. Lapp when class is in session) and MECLABS Managing Director (CEO), Dr. Flint McGlaughlin.
As I sat there taking notes, one subject really stood out to me, and that was testing. It seemed as though many questions were geared towards that. “How big does your sample size have to be in order to know your test results really worked?” “How long should you test for?” “What should you test?” etc…
Testing takeaways
It seems to me that testing still brings up many questions and causes some confusion. Which is why I decided to compile some key takeaways on the matter, with some assistance from our presenters.
1. Get your numbers right
First I’ll say that there is no definite number or sample size when conducting tests. Yes, you do need a minimum number to get valid results, but each case is relative.
“[Sample size] is contingent on how much traffic you have…although it’s much easier and faster to validate a test when you have a bigger sample,” Chuck said.
When you’re a small company you may not have access to test the same amount of traffic Hubspot can, per se, but you still may find the proper number to ensure a statistically valid. As a matter of fact, Director of Editorial Content (a.k.a the person who assigns me to these Bond-style missions), Daniel Burstein, recently explained how you can determine sample size and even broke down the math behind sample size and getting statistically valid test results in his last blog post (Dan is also helping teach a session on test validity at the upcoming Optimization Summit).
But it’s important to get a big enough sample size to ensure that the results you’re seeing aren’t just random chance. If you don’t take the time to discover what that truly is, this can be a big waste of time and money for your company.
2. Test your patience
Seeing a spike in click-throughs, conversions, etc. on a chart is wonderful. I mean, it makes you want to do back flips and give everyone high-fives. But, you have to be careful not to celebrate prematurely. It’s like Dr. McGlaughlin pointed out during one of our meetings, doing this reminds us of that Super Bowl commercial with the monkeys celebrating what appeared to be an increase in sales on a chart, when in fact they had the chart upside down, and they actually did very poorly.
So, how do you avoid getting egg on your face and looking like one of those monkeys? By being patient.
“Some people don’t understand that this is a process. It can be a 3-6 month process,” said Adam. This is especially true for companies with a smaller sample size. “You should test for at least a week to get a cyclical test.”
Test results can fluctuate and while one moment you can be soaring, the next you might also see a huge drop. You have to give the test time to produce the most accurate and valid results possible. And in this process you might have to even re-test, like Chuck said, “your job is never really done. Each time you do a little tweak it’s important to test.”
Not being patient with your tests can end up hurting in the end. And will probably leave you wondering what happened, without learning what really did work and what didn’t.
3. Use every test as a teaching moment
Tests are not conducted to prove someone right or wrong. Their purpose is also not just to measure awesome things happening in your company. You can get a lot more from a test…and that includes pinpointing weaknesses.
“The goal of a test is not to get a good lift, it’s to get a learning,” Dr. McGlaughlin says. Before your head explodes trying to figure out why Dr. McGlaughlin would say getting a good lift is not a goal, hear him out. “If you get an immediate lift, you can’t just be satisfied and walk away…you’re not learning and getting the maximum lift because you’re settling.”
Valid tests results will not only show you where something happened, or what happened, it will show you WHY. The “why” is that part many marketers forget to answer. Tests offer insights on what’s really going on with your audience and how you can improve it for the future. Really taking the time to learn what this data means is the best way to lift your results as high as you possibly can in the future.
But, you must remember, like Adam said, the best way to answer these questions as accurately as possible is by making sure that each test is different enough to actually learn from.
(There was a lot more information covered on testing and landing page optimization at the workshop. You can also be a part of it by attending one of its upcoming tour stops across the country on select dates from June-November)
Related Resources
Marketing Optimization: How to determine the proper sample size
Online Testing and Optimization: ROI your test results by considering effect size
Online Marketing Tests: How could you be so sure?
The Fundamentals of Online Testing online training and certification course
