What is a “Selection Effect” in online testing? And why does it matter?

One of the students from our Online Split-Testing Certification Program recently asked us for some clarification on the meaning of the term, Selection Effect.

If this sounds like a bit of a gray area to you as well, here is the answer we sent him:

Selection Effects is one of the 4 primary threats to test validity and represents the effect on a test variable, such as conversion rate for instance, that occurs as a result of having the sample that is collected be unrepresentative of the actual audience. For example, if you wanted to survey the New York City population about their support for tax reform and took your sample exclusively in the lobby of the Tiffany jewelry store, the results would be skewed due to Selection Effect.

Now, that is an extreme example using a mistake that most of us would be unlikely to commit. In the online world, though, you could have your sample skewed by Selection Effects in a more subtle way. An example occurred for us when we were working with a major news publisher. We had radically redesigned their subscription offer process for the electronic version and were in the middle of testing when they launched a new text link ad campaign from their main website to the electronic product.

This changed the mix of traffic arriving at the subscription offer process from one where virtually all traffic was coming from paid search engines to one where much traffic was arriving from a link internal to their website (highly pre-qualified traffic).

The average conversion rate increased overnight from 0.26% to over 2%. Had we not been monitoring closely, we might have concluded that the new process had achieved a 600%+ conversion rate increase.

This is an example of a Selection Effect validity problem that was not a mistake of design, but one of execution and process control.