Interpreting Results: Absolute difference versus relative difference

A common question that often arises among our Research Partners and audience, even after running many A/B split tests, is: “What’s the difference between relative and absolute difference?”

This is understandable given marketers are not notorious for being data-savvy mathematicians, but let’s face it, interpreting analytics effectively to gain useful insights about your customers is no easy task.

Customer behavior can easily become lost in the math behind campaign results, or worse, overlooked and simply taken at face value.

 

This is troubling given the significant number of marketers who use analytics to improve their messaging (67%) and make decisions (47%) according to the MarketingSherpa 2013 Marketing Analytics Benchmark Report (free automatic download).

In today’s MarketingExperiments Blog post, we’ll take a deeper look at how having a clear understanding of absolute difference versus relative difference can help you interpret your test results.

 

Absolute difference gives you the real number difference between treatments

For the sake of example, we’ll use a simple A/B split test (control and a single treatment) for a hypothetical landing page test and some dummy data to interpret.

 

We ran our test and here are our results. The treatment outperformed the control by an absolute difference of 0.4% and a relative difference of 16.1%.

But what does that mean?

Well, the absolute difference is the difference of two real numbers.

Think of it literally as X – Y.

In this case, the control had a 2.5% conversion rate and the treatment had a 2.9% conversion rate.

2.9% (treatment conversion rate) minus 2.5% (control conversion rate) equals 0.4%

Therefore, our absolute (real number) difference was 0.4%.

We usually look at the absolute difference in conversion rates between a control and treatments to determine the overall difference in performance.

In regard to interpretation, this tells me that the treatment’s conversion rate was literally 0.4% higher than that of the control.

 

Relative difference gives you the scale of difference between treatments 

 

In keeping with our example, we know the treatment outperformed the control by an absolute difference of 0.4% and the literal implications of that.

But what is the real magnitude of the observed relationship difference between our control and treatment that we captured in our data?

This is where relative difference comes in.

Relative difference allows us to understand the comparative ratio of two numbers – often expressed as a percentage – that gives us a direct insight into the true scale of difference between our control and treatment.

In short, for every behavior recorded in the control’s conversion rate, there is 16.1% more of that behavior taking place in the treatment.

 

Here’s an illustration of how relative difference is calculated to help make it a little easier to understand. If we use the data from our test, here’s how it would plug into the formula:

[(2.9 – 2.5)/2.5] * 100 = 16.1%

By understanding relative difference, we can now say the treatment increased conversion by 16.1%.

 

Once you understand it, you own it

Understanding the difference between these two concepts can help you gain a deeper sense of what customers are doing, so you can focus on why.

 

Related Resources:

Validity Threats: 3 tips for online testing during a promotion (if you can’t avoid it)

Testing: Go big, or go home?

Analytics and Testing: Understanding statistics symbols and terminology for better data analysis