Performing a radical redesign can be a daunting endeavor.
You really can’t tell what changes are helping or hurting your conversion rates when you look at the data.
But, what if you were to look into your data beyond conversion rates?
What you find might surprise you.
I say this because recently, we tested a couple of radical redesigns with a Research Partner and had not seen any significant lifts.
So, instead of continuing to try and optimize every element on the page, we decided to change our testing approach.
Return to simple ideas and build from there
We wanted a test that would isolate individual variables, so we could gain transferable insights for future tests.
Control
One simple test idea was to reorder the product listings, so in the control, products were ordered by price in descending order.
Treatment
In the treatment, they were listed by price in ascending order.
Results
Statistically, there was no difference between the two pages based on conversion rates.
Secondary metrics can tell you a lot about your customers
We could have let the test continue to run, but once we reached the required sample size for validation, we decided to shut the test down.
Instead of just declaring the test as a failure, reverting back to the control and moving on, we decided to take a look at the effects the treatment had on revenue.
Our assumption was if the product mix was altered, there would likely be some sort of difference in the revenue also.
Once we looked a little further into the metrics, we discovered a 15% increase in revenue just by starting with the lowest price first, which also validated at a level of confidence of 98%.
Conversion rates are a means to discovery, not an end
Overall, what does this mean?
While there are a few plausible explanations for the increase of revenue, nothing will be conclusive until we run some more tests.
But, this discovery has taught me a valuable lesson about looking past conversion rates to gain customer insights.
So, the next time you have a test and there is no lift in conversion rate, think about discoveries the data offers beyond conversion.
In some cases, that discovery could be a potential ROI you’re leaving on the table.
Related Resources:
Nice work. I read something similar a while ago: They split tested a single product (at the end of the sales letter) and multiple products. The idea behind it was to increase the perceived value of the higher priced products (out of 3 similar products). The conversion rate was much the same, but they received more conversions of the higher priced product, than they did of the lower (originally single product) one.
I think a lot of the time marketers focus too much on the front-end of their business (generating leads/sales), and then forget about the back-end such as the lifetime value of a client and up-selling, down-selling, cross-selling, repeat purchase etc.
Like a lot of internet marketers say, the money is in the list (or the relationship with the list).
Hi Matthew,
Love this. It’s so easy to go with the surface solution, feel good about having done the work and move on, only to be stuck later when things don’t change the way you want them to.
This is a great reminder that digging a littler deeper is almost always worth the time.
-Sarah
Content Marketing, Union Metrics
Fine Makers of TweetReach and Union Metrics for Tumblr