Focus Groups Vs. Reality: Would you buy a product that doesn’t exist with pretend money you don’t have?

Would you believe a focus group over reality? Of course not. Experienced researchers, including those that regularly work with focus groups, know how difficult it is to orchestrate a focus group study that would produce realistic predictions.

“Calibration” is the name of the game, but there might be little or nothing to calibrate by if what you are studying is radically different from any previous available studies. Or if the study finds a problem with your boss’s favorite squirrel (the name we gave to your boss’s bad idea in Part 1, the idea that you can’t quite change his mind on)…

A cloudy crystal ball

When you are developing a new product, you have little choice. My team was recently involved with a Web design company whose client was a major CPG (consumer packaged goods) manufacturer that was rolling out a wholly new product category.

Extensive focus group research performed by an experienced agency delineated a target market (complete with size, demographics, and psychographics), an acceptable price point, and a sub-set of product benefits that would be received especially favorably as part of a marketing message.

The manufacturer is a highly sophisticated investor in new products, and found the business plan for this new idea sound and valid. Yet, out of the gate, the product didn’t perform. The intended customers simply weren’t buying, even though the focus group predicted they would.

Of course, there are many moving parts here: were the infomercials poorly produced? Was the landing page sub-optimal? Perhaps. However, all of the marketing and sales messaging was developed strictly based on focus group preferences.

Does this mean that the focus group research was wrong? One thing I am certain about is that the research was performed to the highest standards. Why didn’t it predict what would really happen?

Would you buy a product that doesn’t exist with pretend money you don’t have?

This experience underscored the fundamental problem with using focus groups to decide what people are willing to buy. When it comes to purchase decisions, we cannot expect consumers to be able to imagine a buying situation so perfectly that their make-believe decision would be identical to a real-life one.

Again, I didn’t just discover this. Marketers have known this since they started using focus groups. So, what’s new?

“The real voyage of discovery consists not in seeking new landscapes but in having new eyes.” – Marcel Proust

I suggest that with interactive Internet media (Web, email, social media, etc.), focus group testing can be useful to get an approximate idea of sentiment and preferences, but real-life behavioral testing is what produces the greatest return on the time, effort, and money spent on the research.

Based on preliminary focus group research (again, I am not discounting it entirely), a small batch of the physical product can be produced (if the product is entirely digital, this is a no-brainer) and offered online without major investment in ad media.*

Of course, you will still need to drive traffic to the site. Paid Search is a measurable way to do just that, since you have full control of spend and real-time reporting of how that spend performs (or under-performs) to test and manage your ad copy. Likewise, your landing pages provide an opportunity for real-time testing on real incoming traffic of potential customers.

Real time, real-world data

Rather than ask these people to imagine a purchase situation, we can observe them making an actual purchase. While asking them about what they think, feel, or are “likely” to do is impossible, we can measure what they actually do in a rather fine detail. We know whether they buy or not; we know whether they return to the site or not; we know even which step of the shopping cart is the “leak” in the funnel.

There are certainly some blind spots. For example, we can’t tell why a particular customer bought, and another didn’t. We also don’t know (depending on how long the test runs) if a customer is planning to come back to buy later. We don’t know if something outside of our own efforts (ads, website copy, etc.) moved the customer to buy (or not buy).

Do vs. If

Behavioral data is necessarily aggregated above the level of individual preferences and sentiment. However, with some help from statistics, it tells us with the requisite level of confidence what consumers do or do not want (as opposed to would or would not want if…).

Making the business case for testing

On Wednesday, we’re devoting an entire web clinic to not just our latest online marketing optimization discoveries, but a behind-the-scenes look at how one marketing manager implemented a culture of testing across her enterprise organization.

* Note that it is against FTC rules to sell a product that doesn’t exist. The more unscrupulous marketers go as far as testing a product in Paid Search ads before it’s ever been produced to gauge demand, and then refund consumers money if too few buy.

Related research

Never Pull Sofa Duty Again: Stop guessing what your audience wants and start asking

A/B Split Testing

MarketingExperiments Methodology