Online Testing: How a pop-up chat test increased conversion 120%

In a world where content is king, sifting through Internet clutter postured as content is generally accepted as a daily task for many of us.

The perpetual endeavor of distinguishing actual valuable content in the minds of an apathetic populace numbly navigating the Internet landscape without any concern for your KPIs can feel overwhelming at times.

Due to this harsh reality, many marketers turn to the pop-up window to solve the issue of engagement.

Their hope rests on the notion that an interruption will somehow distinguish their content as valuable, often with the opposite effect as content is likely to be ignored the second a pop-up is triggered.

Yet, the use of the pop-up window has proliferated across the Web and has become an often valuable tool for many skilled marketers. However, analysis of this webpage element yields one fundamental question:

“Is the pop-up window an effective option when attempting to gain a visitor’s attention?”

 

Don’t speculate, test

Consider a recent test performed by MECLABS. The team attempted to optimize a landing page of a well-known Fortune 500 B2B company. Traffic for this landing page was driven by paid search, PPC ads.

The landing page’s objective was to build the value of a free trial offer in order to increase the amount of leads submitted by visitors.

 

Control page with pop-up 

 

The page provided visitors with three options which they could use to speak to a qualified representative:

• Click on a chat box, which automatically pops up after an incremental amount of “time on page”
• Fill out a lead capture form which notifies a sales rep to contact them
• Call a phone number

Because the chat box converts at a significantly higher rate than the other two options, the MECLABS research team was tasked with optimizing the page to increase the amount of chat clicks.

In order to accomplish this, we set up a test where the pop-up was replaced with an actual static image of a chat box placed on the right side of the page.

 

Treatment page with static chat box 

(*Please Note: We have anonymized the landing pages to protect the competitive advantage of our Research Partner. – Ed.)  

 

Visitors were encouraged to click on the image of the chat box, which then opened an actual chat box in a new window.  Other variables that changed on the page were:

•  Value copy associated with the process of contacting a representative was added above the form fields
•  Location of the lead capture form on the page, which was pushed farther down the page  to accommodate the increased value copy)

Would a static page element that doesn’t actively vie for a visitor’s attention perform better or worse than a pop-up?

 

Results

The data for this test yielded very interesting results.

 

The treatment resulted in a dramatic lift of 120.60% in chat clicks over the control at a 99% level of confidence.

However, this did come at the expense of lead capture form submissions and phone calls. The treatment resulted in a decrease of form submissions by 60.54% at a 99% level of confidence, and a 33.64%  decrease in phone calls at an 89% level of confidence.

After performing an in-depth data analysis, it was determined the implementation of this treatment was projected to increase revenue by $868,591 over the course of a year (261 days, weekends excluded).

So the question arises again, “Is the pop-up tool an effective option when attempting to gain a visitor’s attention or capture leads?”

Clearly the answer in this case is “no.”

That being said, will removing your pop-up chat box and implementing it as static page element result in dramatic results for your business?

Well, therein lies the beauty of testing, as it’s the only way you’ll know for sure.

 

Related Resources:

Landing Page Optimization: 6 common traits of a template that works

Landing Page Optimization: Simple pop-up overlay increases conversion 63%

Landing Page Optimization: Simple color change increases conversion 10%