Landing Page Optimization: How The New York Times generated a 1,052% cumulative conversion gain

David,
Thanks for your question. I asked Bob Kemper, Director of Sciences, MECLABS for some more information, and here’s what he had to say…

Hello David,

That’s a great question. While that test series with the New York Times took place with the NY Times Electronic Edition, considerably before their recent experimentation with the paid subscription model, still your point is an important one. The question is, “How can we minimize the considerable threat that History Effect can present to the accurate assessment of optimization testing gains, particularly when they take place over a substantial period of time?”

Our ability to arrive at the stated cumulative conversion gain with reasonable confidence is a product of a couple of specific experimental design practices that serve to limit the level of error associated with such multiple stage test sequences. In particular…

1. All of the tests in the experimental sequence were conducted using a random sampling simultaneous split testing methodology—and each was oversampled. Consequently, the confidence intervals of the constituent tests in the sequence were comparatively narrow.
2. At each stage in the series, the highest-performing treatment from the prior stage was used as that test’s Control—and there was a comparatively brief time interval between successive test stages.

These experimental design characteristics are integral to conducting such extended-term, multi-stage optimization programs as this 16 month series.

That said though, there are, in fact, a number of sources of error. In particular, each of the tests in the sequence is subject to its own corresponding statistical margin of error. When the cumulative impact of a series of tests is computed, as is the case here, the aggregate margin of error for the test series expands as well. So, while 1,052% does constitute the best ‘point-estimate’ representation of the cumulative performance impact of the test sequence, it would be more mathematically and statistically sound to combine that figure with the corresponding aggregate statistical level of significance for the complete test sequence and present it as a range estimate at a specified level of confidence. So why is the single point-estimate used in this (and others) research brief video? We have found over time, that for our primary constituency of professional marketers, the overwhelming majority prefer the ‘point’ form for its brevity and efficacy at conveying the central message (even as they are aware that there is a corresponding margin of error). Consequently, we typically leave the range-estimate form for our own internal publications and for the footnotes or endnotes of any technical or other research journals in which they may appear.

Thanks once again for your insightful question, David. I hope you’re finding MECLABS’ research useful, and I wish you much success in your own optimization efforts.

All the best,

Bob Kemper
Director of Sciences, MECLABS