Same Discount, Different Results

Other Important Variables in Discount Experiments

1

#1. Different Regular Prices

Previously on iOS, we saw a whopping 40% increase in new subscription revenue through a price discount experiment. We learned a lot back then, which you can read about in [Are Discounts Always a Win? — Finding the Optimal Discount Strategy for App Subscriptions]. Building on those learnings, we confidently rolled out an experiment on Android with Variant 3 (30% off monthly subscriptions), skipping Variants 1 and 2. We targeted the same user segment and offered the same discount rate. But the results were quite different.

While the variant won with 99% statistical significance in both cases, the extent of the improvement differed. iOS saw a 40% boost, whereas Android only saw 18%. This could partly be because the features offered on the two OSs are slightly different product-wise, but we identified the difference in the original price as the biggest variable.

The regular monthly subscription price on iOS was 8,500 KRW, while on Android, it was 5,900 KRW. Since Android already had a relatively lower regular price, a 30% discount might have been less appealing in itself. The absolute size of the discount likely played a role too—on iOS, it looked like a drop of about 3,000 KRW ($3.00), whereas on Android, it only looked like 2,000 KRW ($1.50).

An 18% improvement in new subscription revenue was definitely a valuable result, but I couldn't help feeling a bit disappointed—if we had set up multiple variants with slightly different discount rates, we might have achieved a much bigger win. I need to make sure I never overlook the fact that "the starting point of a discount is the regular price it's applied to."

#2. Different Durations of Opportunity

Our previous iOS experiment brought in 1.4x the revenue, but we still saw room for further improvement. Specifically, quite a few users ended up subscribing at the "regular price" even after being shown the discount offer. At the time, the discount was a one-time offer—if they closed the purchase window, the chance was gone. The fact that many users later subscribed at the regular price meant we might have offered the discount for too short a time. (Or, it could mean we offered it too early. We plan to run a separate experiment for that hypothesis.)

What if, instead of a one-time pop-up, we provided an entry point they could return to for a certain amount of time? Wouldn't that generate a lot more discounted subscription revenue? Common sense says that longer exposure can't hurt revenue, but we had to worry about cannibalization—losing potential "regular price" subscribers. We also had to consider that a one-time offer might actually create more urgency ("Close this and it's gone!") and drive more discounted subscriptions. That's why we decided to put this hypothesis to the test as an "experiment."

Here, the variable was "how long to keep that entry point visible." In other words, "how much time to give the user to think it over." It could be 24 hours, 1 hour, or 10 minutes. To minimize trial and error, we looked at growth case studies from other companies and ultimately decided to give them exactly "10 minutes" of deliberation time.

The identical discounted purchase screen is shown at the exact same moment, but for the variant, a 10-minute countdown starts. The variant also has an entry point in the top right corner, allowing users to return even if they leave the screen.

The result?

This experiment yielded an additional 10% improvement in revenue (with 99% statistical significance). Compared to the control group of our previous discount experiment, which had no discount at all, that translates to an overall improvement of 1.54x.

What was even more interesting than the improvement itself was the average number of times the discounted purchase screen was shown to the variant group. Since the control group was a one-time exposure, their average screen view count was "1." For the variant, which allowed users to revisit the screen for 10 minutes, the average was only "2." Given that the first view is forced, an average of "2" means users voluntarily tapped the entry point to return just once.

Diving deeper into the data, when we set the cohort to "users who eventually subscribed," the average number of visits was less than 2 (1.7 times). Meanwhile, for the cohort of "users who never subscribed," the average was higher than 2. "Two" is already a low number of visits, but even that was inflated by the group of users who just hesitated without ever subscribing. This meant that users who actually "subscribed at a discount" usually went for it almost immediately upon the first exposure.

A statistically significant 10% revenue improvement is great, but realizing that "giving a longer discount window" isn't a hugely impactful variable was even more valuable. When it comes to discounts, whether you give them "an extra 30 minutes or an extra 24 hours" probably doesn't matter much. Instead, it seems like a better idea to run follow-up experiments on varying the "timing" of when the discount is offered.

We're going to hit pause on our fast-paced discount experiments for a bit and run a regular price test for the first time in quite a while.

If this ends up changing our regular price, we'll have to plan our discount experiments all over again from scratch. But I think "knowing that we'd have to start over from scratch" is perhaps the most precious lesson we've learned from all our discount experiments so far.

To the discount experiment series, which was full of learnings in all sorts of little details,

Goodbye for now!