Have you worked on a project where the data reporting was less than ideal?
An odd conversion rate here, or something fishy about the funnel setup there. The momentum builds and before you know it, a lack of data integrity has become a confidence killer.
Sadly, confidence killers do occur more than I would like when working on our Research Partners’ websites, which brings us to a greater problem …
How can you optimize a website if the integrity of your data is highly suspect?
In today’s MarketingExperiments Blog post, I wanted to share four simple tips to help you gain confidence in your metrics reporting that you can use to aid your testing and analytics efforts.
Tip #1. Earn analytics access if you can, and fight for it if you can’t
This one may be a no-brainer, but in some cases, access to a company’s analytics is limited even to the research team that relies on the information to improve performance.
The big problem with this limitation is it can completely undermine the good faith involved in any project. So, how can you avoid this issue in the first place?
A simple answer is to try and negotiate access to any and all information you may need for your project in advance and pour over those analytics with your data analyst. With any luck, you’ll be able to spot any inconsistencies before you start testing.
I’ve seen issues ranging from two sites incorrectly using the same UA code, which is the tracking ID Google Analytics uses to tie a site to the analytics platform, which renders all traffic numbers inaccurate. I’ve also witnessed funnels with astronomical conversion rates caused by an overabundance of test orders, and everything in between.
Without gaining access to really dig into the metrics, what guarantee is there that you would ever get to the root of the issue?
But, if you can’t receive access no matter what you try, here are a few things you can do.
- Request reports for the areas of the site you are working on in advance at the beginning of your project.
- Analyze any data you can access to establish a baseline, and identify validity threats that could endanger the project.
If you spot inconsistencies, make sure to notify business leaders about those problems to help you effectively manage testing expectations. If you are comfortable with the data, agree on a set of reports and KPIs the project will be judged on.
When testing or launching a major refresh of a page, it is also important to agree on a set schedule of reporting so you are able to check the fidelity of the data and catch anything that could pop-up in a reasonable timeframe. I would suggest checking the data within two days of the change and every other day from that point in time going forward.
Tip #2. Treat quality assurance as an integral component of every project
I cannot stress how important a good QA process is in ensuring the data you are basing your recommendations on is accurate. While it may not be the most glamorous part of the project, it is often one of the most important components to boosting accuracy and mitigating risk.
How do you ensure your funnels are tracking properly or that a test was set up correctly?
It can be as simple as setting up a few test scenarios and recording the selections you make, pages you visit, buttons you click or events that should fire, and then ensuring the analytics platform matches the data you entered as you recorded it. This should be performed in a staging environment first, and then duplicate what you have done in a production environment.
Tip #3. Set up tracking in another analytics or testing platform
What analytics platforms do you use to track your online sales and marketing efforts? Do you use multiple platforms? Do these platforms integrate with backend sales data?
If you answered “yes” to any of those questions, then you should consider setting up redundant tracking across multiple analytics platforms or within the same platform.
While it’s not realistic to think everything will match up perfectly, this can help your team decide on an acceptable margin of error between platforms that can be monitored for inconsistency.
Also, make sure important funnels, events and pages are tagged consistently between platforms and checked for accuracy.
A caveat I would offer here is to be wary of the different definitions of metrics and how they are calculated across analytics platforms because a specific metric in one platform may not be defined the same across all of them. Try to find the most similar metrics in definition and base your comparison of platforms on that selected metric.
Using multiple platforms for tracking can also help your long-term tracking engagements where it helps to have another set of eyes on the data.
Tip #4. Use dual control A/B split tests
This final tip is specifically for digital marketers who want to cross check their reporting during test periods.
Have you ever had an unbeatable landing page with the suspicion it was the metrics platform and not the page itself that was proving unbeatable?
At MECLABS, we call this an instrumentation effect.
Perhaps you thought the traffic split was unfair or somehow the experience visitors encountered of being placed in the test caused your treatments to fail. The simple way to alleviate your fears is by running a dual control A/B split test where both pages are exact duplicates of one another.
You can also set up different goal funnels for each page to track other primary or secondary metrics that you may be interested in. If the test ends up being a wash and neither page significantly outperforms the other, then your tracking can at least be deemed consistent for testing purposes.
Don’t leave your analytics to the mercy of chance
My whole point here is with a little planning, the confidence in your metrics will not be reduced to the mercy of chance. The simple four tips above will help you improve confidence in your own metrics and may even help you spot some issues you may not have known existed.
In short, let someone else’s customers suffer from testing strategies built on coin tosses or arm wrestling contests because serving yours with confidence is so much more rewarding.
Do you have any other tips or comments that help you make sure your metrics are accurate? Please leave them in the comments section below.
Related Resources:
Web Tracking and Online Testing: Come to the dark side, we have cookies!
Metrics and Analytics: 4 questions every marketer should ask their data analysts
Marketing Analytics: Should you use a daily or aggregate method to validate A/B test results?
