We talk a lot about A/B testing here on MarketingExperiments. What we don’t usually talk about is A/B testing for the mobile web…especially testing within mobile apps.
I thought we should change that. As I was scouring the web looking for mobile A/B tests, I found this 2-year old video by Amazon.
Apparently, Amazon Web Services (AWS) at one point had an A/B testing feature that is now closed.
When they had testing, however, one app developer used it extensively and shared their experiences in a promotional video for the feature on Amazon. The developers were behind the game Air Patriots. Russell Caroll was the Senior Producer for the game and Julio Gorge was the Game Development Engineer. The game is a kind of aerial take on the classic tower defense game genre.
Granted, this was a promotional video, but the content still speaks for itself. These guys had (and still have by the looks of it) a fairly successful mobile app and they ran some successful tests. It’s a great starting place for what you can test in your mobile app.
By the way, while Amazon has shut down its A/B testing feature, there are many other tools for testing mobile apps that will accomplish the same thing the developers talk about in the video.
Test #1: What is the impact of ads on customer experience? (1:34)
The team first tested the impact of ads on their customers. They wanted to make sure the ads did not harm the customer experience, so they tested a single ad in the main menu near the bottom of the screen.
They found that the ads didn’t affect customer retention. This meant they could insert ads and generate more revenue without hurting their customers.
Test #2: Will in-game ad placement affect customer retention? (2:56)
In the second test, the team put ads in the game screen.
In both the first and second tests, a small “X” was placed in the ads that customers could tap to hypothetically dismiss them. When they tapped, a pop up came up telling customers they could eliminate ads with any purchase in the game’s store.
In this test, there was again, no impact on customer retention, but there was a statistically significant increase in revenue.
Test #3: Simple GameCircle icon test (4:20)
In this test, the team wanted to know if placing an icon to GameCircle (Amazon’s game stats and leaderboards portal) would improve performance.
It’s not clear which icon won, or even why this particular test was useful for the team, but they did get a favorable result. The lesson they wanted to drive home was that simple changes like icons can make a difference. We’ve also found that to be the case in a large number of our tests on MarketingExperiments.
Test #4: Does game difficulty affect revenue? (4:58)
In this fourth test, Caroll made a mistake. He accidentally changed the game difficulty to make it about 10% harder. As a result, every important metric the team measured tanked.
The team fixed it as quickly as possible, but it gave them an idea.
What would happen to revenue if they made the game easier?
So they ran a test with five treatments – the control and then four difficulty levels that were easier than the control.
It turned out that the easiest difficulty performed the best. Making it easier resulted in the players playing 20% longer and revenue increasing 20%.
Test #5: When is the best time to have push notifications for re-engagement in inactive players? (7:43)
The team then tested a push notification that offered inactive players and incentive for picking the game back up.
They wanted to know when the best time to send the notification would be. So they tested a few different variables and found that the best time was three days after the last game play.
They also found that sending the notification seven days after the last game play negatively impacted their performance metrics.
With these five tests and probably a few more taking place off the record, the team was able to develop a high-quality app for their customers and steadily increase their revenue. At the end of the video Carol gives a few key takeaways for marketers who are A/B testing their apps.
You might also like:
Email Research Chart: Email opens trends on mobile devices in 2015
Mobile Marketing Chart: Amount of revenue from the mobile channel, by merchant type
