Interpreting Results: Absolute difference versus relative difference

4

A common question that often arises among our Research Partners and audience, even after running many A/B split tests, is: “What’s the difference between relative and absolute difference?”

This is understandable given marketers are not notorious for being data-savvy mathematicians, but let’s face it, interpreting analytics effectively to gain useful insights about your customers is no easy task.

Customer behavior can easily become lost in the math behind campaign results, or worse, overlooked and simply taken at face value.

 

This is troubling given the significant number of marketers who use analytics to improve their messaging (67%) and make decisions (47%) according to the MarketingSherpa 2013 Marketing Analytics Benchmark Report (free automatic download).

In today’s MarketingExperiments Blog post, we’ll take a deeper look at how having a clear understanding of absolute difference versus relative difference can help you interpret your test results.

 

Absolute difference gives you the real number difference between treatments

For the sake of example, we’ll use a simple A/B split test (control and a single treatment) for a hypothetical landing page test and some dummy data to interpret.

 

We ran our test and here are our results. The treatment outperformed the control by an absolute difference of 0.4% and a relative difference of 16.1%.

But what does that mean?

Well, the absolute difference is the difference of two real numbers.

Think of it literally as X – Y.

In this case, the control had a 2.5% conversion rate and the treatment had a 2.9% conversion rate.

2.9% (treatment conversion rate) minus 2.5% (control conversion rate) equals 0.4%

Therefore, our absolute (real number) difference was 0.4%.

We usually look at the absolute difference in conversion rates between a control and treatments to determine the overall difference in performance.

In regard to interpretation, this tells me that the treatment’s conversion rate was literally 0.4% higher than that of the control.

 

Relative difference gives you the scale of difference between treatments 

 

In keeping with our example, we know the treatment outperformed the control by an absolute difference of 0.4% and the literal implications of that.

But what is the real magnitude of the observed relationship difference between our control and treatment that we captured in our data?

This is where relative difference comes in.

Relative difference allows us to understand the comparative ratio of two numbers – often expressed as a percentage – that gives us a direct insight into the true scale of difference between our control and treatment.

In short, for every behavior recorded in the control’s conversion rate, there is 16.1% more of that behavior taking place in the treatment.

 

Here’s an illustration of how relative difference is calculated to help make it a little easier to understand. If we use the data from our test, here’s how it would plug into the formula:

[(2.9 – 2.5)/2.5] * 100 = 16.1%

By understanding relative difference, we can now say the treatment increased conversion by 16.1%.

 

Once you understand it, you own it

Understanding the difference between these two concepts can help you gain a deeper sense of what customers are doing, so you can focus on why.

 

Related Resources:

Validity Threats: 3 tips for online testing during a promotion (if you can’t avoid it)

Testing: Go big, or go home?

Analytics and Testing: Understanding statistics symbols and terminology for better data analysis

You might also like

4 Comments

  1. Jon Chacko says

    Thank you for walking through this. I noticed that your equation lists Control Conversion minus Treatment Conversion divided by Control Conversion, BUT your illustration orders it Treatment Conversion minus Control Conversion divided by Control Conversion.

    Can you explain that?

    1. Shelby Dorsey says

      Hi Jon. Great catch. I do see that we flip-flopped between the equation and the illustration on whether it’s Treatment Conversion Rate MINUS Control Conversion Rate or whether it’s Control Conversion Rate MINUS Treatment Conversion Rate. The reality is that as long as you use the same denominator (the Control Conversion Rate) the ABSOLUTE DIFFERENCE between Treatment Conversion Rate and Control Conversion Rate would be the same and so too would the RELATIVE DIFFERENCE albeit negative or positive. So, in both cases we end up with a 16% difference but when subtracting the Treatment Conversion Rate from the Control Conversion Rate and then dividing it by the Control Conversion Rate (as seen in the equation) we end up with -16% difference as opposed to the positive 16% difference which results when using the order shown in the illustration. Ultimately, it comes down to which one you’re comparing. In the case of the -16% difference, we’re saying the Control had a 16% decrease compared to the Treatment. However, in most cases you’ll want to compare Treatment performance to the Control which makes the order shown in the illustration more appropriate: (Treatment Conversion Rate MINUS Control Conversion Rate)/Control Conversion Rate = 16% increase. Hope that helps.

Leave A Reply

Your email address will not be published.