All online testing platforms are not created equal.
This is because no organizations and testing programs are created the same. You need a testing platform that fits the individual needs of your organization.
With the numerous tools available, it can seem like a daunting task. But, keeping a few considerations in mind could help you significantly narrow down the options.
In a recent MarketingSherpa (our sister company) article, “A/B Testing: How a landing page test yielded a 6% increase in leads,” Todd Barrow, Development Manager, MECLABS (the parent company of MarketingExperiments), suggested four factors you should consider when choosing a testing platform for your organization: cost, IT requirements, support and technology ecosystem.
Because a testing platform is so important to an organization’s testing program, I spoke with Todd and Jessica McGraw, Senior Technical Manager, MECLABS, to learn more about these four factors.
Factor #1: Cost
The cost of online testing platforms ranges from free to thousands of dollars. But, even free tools come with hidden costs.
That’s why you must consider both the cost of the software itself as well as your own costs of implementation. The time and resources of both the marketing and IT departments are limited.
The less expensive tools may come with less support, and would then require more involvement from your development team.
“For instance, Test&Target™ costs a lot, but you get a lot of support, and they’ll tell you exactly how to set everything up. Whereas, say, with Google [Website Optimizer], you get no support, and you have set everything up. So, either way, there’s a cost in either the support itself or for some developer or expert,” explains Todd.
Factor #2: Support
There is a wide variety of support levels for testing platforms. Many free platforms offer little to no support, outside of basic instructions on their websites. Some expensive platforms provide unlimited support, which, in many cases, includes a personal account rep.
A number of platforms will even give you a choice of different support packages, so you can base the support level on your organization’s individual needs.
Based on your internal IT resources, you may only require a basic support package with delayed email support. If you have little to no resources available, you may need a platform that provides an account manager to have the dedicated support you’ll require.
And, if your IT team has the resources to be heavily involved, a free tool with step-by-step instructions, FAQ and forums could work well for you. You might never miss having a contact person if you have a highly involved IT department.
Todd explains his decision based on support, “When I’m looking at what platform I want to choose and I have an idea that one of the free ones will require an expert, I find out if I have a resource like that or a resource that has the time I need to run the number of tests I want. Then, if I don’t, I would choose different option, one that only requires Dev [development by the IT department] one time or one that will walk you through everything and hold my hand.”
Factor #3: IT requirements and resources
As you can see from the support factor, another big factor is based on what IT resources you have, and what they’re capable of doing.
Your first obstacle could be whether or not Marketing has the authorization to even make those changes.
“There are a lot of companies out there that their IT teams own everything, and they won’t allow you to install some weird tool that will scan your entire site or let you make changes because they want to control it,” Jessica says.
On the other side, you could be fortunate to have the IT resources and buy-in to give Marketing complete control. Jessica described an example using a MECLABS Research Partner with which she worked.
“They got their IT on board, and the IT team went through and installed the code that was needed on every single page throughout their site so Marketing could control everything within the (testing platform).”
This is why it’s so important to bring IT into the conversation before a testing platform is selected. It might require some compromise on both sides to find the right tool, but you won’t know until you determine the IT resources available to Marketing.
Jessica continues, “Then, sometimes, [companies’ IT teams] don’t have the resources to go in and make every change.”
You will need to know if the testing platform is something Marketing can own completely, or will you have to rely on IT for each change.
If this is the case, and you plan to run a significant amount of tests, then a platform that requires extensive work from IT for each test is probably not right for your organization.
You also have to consider the level of expertise the tool will require. Some might provide a “What You See Is What You Get” (WYSIWG) editor, so that even a marketer with limited Web development knowledge can do basic layout changes. Others might need complex HTML coding for each change on each page.
Factor #4: Technology ecosystem
The last factor takes into account how the potential platforms will impact your current Web platform. While you don’t have to worry too much about a platform being completely incompatible, you do need to consider whether you have the resources to change the system so that your technology ecosystem works in harmony, says Todd.
Jessica elaborated, “It’s how much effort you want to put into making that platform work with your current [Web] platform — which is part of your evaluation process — because there’s always another platform out there that is easier to install or set up.”
Todd brought up another Research Partner example based on Jessica’s comment.
“They had a specific platform and already had their own account. But, to do the test we wanted to do, we had to change the way they used the testing platform, and because their IT resources were limited, they had a hard time doing that so we could then do our test.”
Have a thorough conversation with IT
It all really comes down to the goals of your testing program and whether you have the resources you need to implement them. And, you won’t be able to determine those things without sitting down with your IT team.
“If you can narrow it down to two or three (testing tools), that’s fantastic. Because then you’ve put enough time and effort into it to really know what you’re looking for in the testing tool. Granted, you haven’t necessarily seen the IT side of it, but now I know the feature list that’s important to Marketing,” Jessica says.
Todd adds to this, “Without having that clear objective of what you want to get out of it, it makes for a more random conversation.”
With your objective and features list in hand, you can then sit down with your IT team to determine what’s possible and what’s not based on the resources available. Then, you can work out what online testing program will best fit your company’s specific needs and wants.
For more information about specific platforms, visit the MarketingExperiments blog quick guide to 8 testing platforms.
Related Resources:
Optimization 101: How to get real results from A/B testing
Online Testing and Optimization: ROI your test results by considering effect size
Great Article