In this video, Dr. Flint McGlaughlin, Managing Director and CEO, MECLABS, shares how The New York Times and MECLABS worked together to generate a 1,052% landing page conversion gain during 18 months of experimentation …
00:28 – Maximizing the efficiency of the sales process
1:19 – Experiment background
2:28 – Test #1
3:56 – Test #1 results
4:27 –Test #2
5:02 – Test #2 results
5:30 – Test #3
5:52 – Test #3 results
5:58 – Test #4
6:05 – Test #4 results
7:31 – Test #5
7:44 – Test #5 results
7:56 – Cumulative results of the experiment
15 years of marketing research in 11 minutes
Landing Page Optimization: How to plan a radical redesign so you get a lift AND a learning
MarketingExperiments Research Directory
Can these numbers tell the real story when the testing was done over 18 months? I think over this time the NYT was lowering the number of free articles per month that a visitor can read without subscribing and promoting very heavily the paid subscription.
Thanks for your question. I asked Bob Kemper, Director of Sciences, MECLABS for some more information, and here’s what he had to say…
That’s a great question. While that test series with the New York Times took place with the NY Times Electronic Edition, considerably before their recent experimentation with the paid subscription model, still your point is an important one. The question is, “How can we minimize the considerable threat that History Effect can present to the accurate assessment of optimization testing gains, particularly when they take place over a substantial period of time?”
Our ability to arrive at the stated cumulative conversion gain with reasonable confidence is a product of a couple of specific experimental design practices that serve to limit the level of error associated with such multiple stage test sequences. In particular…
1. All of the tests in the experimental sequence were conducted using a random sampling simultaneous split testing methodology—and each was oversampled. Consequently, the confidence intervals of the constituent tests in the sequence were comparatively narrow.
2. At each stage in the series, the highest-performing treatment from the prior stage was used as that test’s Control—and there was a comparatively brief time interval between successive test stages.
These experimental design characteristics are integral to conducting such extended-term, multi-stage optimization programs as this 16 month series.
That said though, there are, in fact, a number of sources of error. In particular, each of the tests in the sequence is subject to its own corresponding statistical margin of error. When the cumulative impact of a series of tests is computed, as is the case here, the aggregate margin of error for the test series expands as well. So, while 1,052% does constitute the best ‘point-estimate’ representation of the cumulative performance impact of the test sequence, it would be more mathematically and statistically sound to combine that figure with the corresponding aggregate statistical level of significance for the complete test sequence and present it as a range estimate at a specified level of confidence. So why is the single point-estimate used in this (and others) research brief video? We have found over time, that for our primary constituency of professional marketers, the overwhelming majority prefer the ‘point’ form for its brevity and efficacy at conveying the central message (even as they are aware that there is a corresponding margin of error). Consequently, we typically leave the range-estimate form for our own internal publications and for the footnotes or endnotes of any technical or other research journals in which they may appear.
Thanks once again for your insightful question, David. I hope you’re finding MECLABS’ research useful, and I wish you much success in your own optimization efforts.
All the best,
Director of Sciences, MECLABS
That’s amazing. I’m very interested in the way you convinced people of IT to take such big steps forward. There are some great political lessons to be learned from a proces like this. Did it take long to convince the DMU to take certain steps in changing the pages and how analytical schema’s of codes were implemented etc.?
As I’m sure you understand, it would not be appropriate to publicly discuss the internal deliberations of a Research Partner.
However, given your interest, I think you may find this blog post helpful — Website Optimization: Why the Marketing-IT relationship is critical and 7 tips to make yours stronger.