The Subscription and Retention Balance Game
2x improvement in subscription conversion vs 10% improvement in D1 retention
Through our first subscription prompt experiment, we learned that nudging users to subscribe early on significantly increases the subscription conversion rate, but also drops D1 retention. (Previous post: The Importance of the First Subscription Prompt Considering Retention — Optimizing the timing of the onboarding purchase screen) Between the "control group with no drop in retention but a low subscription conversion rate" and the "experiment group with greatly improved subscription conversion but dropped retention," we initially sided with the control group. There was a simple math comparison, but more importantly, we had the confidence that we could discover a more balanced winner in subsequent experiments. We decided to plan an experiment group that would increase the subscription conversion rate while preserving retention, even if we nudged for a subscription early on.
Experiment 3
2nd to 3rd week of October 2023
Our previous first and second experiments were about when to show the purchase screen. Through those experiments, we found our own most powerful timing. However, the problem discovered in those experiments was the drop in retention, and the issue we defined for "why do they leave the app?" was "because users don't know how to exit the purchase screen." Therefore, for our third experiment, we decided to test what exit method would be optimal.
The result was unexpected.
Did we actually make them want to exit more? The subscription conversion rate was cut in half in the experiment group. And fascinatingly, D1 retention improved by 5-7%. In the last experiment, the subscription conversion rate doubled and D1 retention dropped by 8%, but this time we got the exact opposite result.
[Exp 1-2] When it was hard to exit the purchase screen, subscription conversion went up and D1 retention went down.
[Exp 3] When it became easy to exit the purchase screen, subscription conversion went down and D1 retention went up.
Hmm... Then...
What if exiting the purchase screen was neither easy nor hard? Would both retention and subscription conversion go up?
It felt like we could find a balance point if we tried a few more times.
Experiment 4
1st to 2nd week of November 2023
This time, we proceeded with the fourth experiment by preparing a purchase screen that users could exit moderately easily. As time passed, the shape of the control group's purchase screen also changed due to other experiments (the broadly smiling face of Emma disappeared). So, to check that change itself as a new variable in this fourth experiment, we set up Experiment Group 1 with the exact same exit conditions as the first experiment (only the purchase screen changed), and set up Experiment Group 2 where exiting was made slightly easier. For Experiment Group 2, we applied some changes to the close button exposure logic.
The results this time were beautiful.
Experiment Group 2, which allowed users to exit moderately(?) easily, achieved an increase in the subscription conversion rate without a drop in D1 retention.
Experiment Group 1, which had the same exit conditions as the first and second experiments, generated a larger increase in subscription conversion than Group 2, but suffered a drop in D1 retention.
Wow, we finally found the balance point..! (Honestly, I started this without even thinking such a 'balance game' would exist..)
After four rounds of experiments, we found the optimal spot. A strange sense of relief washed over me.
But at the same time, a final surge of ambition rose up.
Group 1 showed a 40–50% improvement in subscription conversion; couldn't it win in a showdown against the drop in retention (5–10%)?
When we first compared the value of the subscription conversion rate and D1 retention, we looked at their quantitative values. Since D1 retention also leads to ad revenue, we converted it into revenue and pitted it against the incremental subscription conversion rate. The formula calculates different amounts depending on how many users come in every week, what the LTV of each user (subscribed user, free user) is, etc. To make this comparison easy this time around, our team's DA Leo built a tool for us. When we used it to calculate the incremental subscription conversion rate and the retention drop for Group 1 in the 4th experiment, it looked like this.
The calculation showed that the revenue increase from the higher subscription conversion was larger than the revenue loss from the lower retention. Even so, after much discussion, we chose Group 2 instead of Group 1. If we only considered the quantitative value, picking Group 1 was the right move, but we had our own final guardrail separate from that.
The Final Guardrail
Even if users leave our app, what impression do they have when they leave? This significantly changes the likelihood of them coming back and the brand image of our product that spreads through word of mouth. In other words, even if D1 retention drops by 5% in both cases, the impact of leaving out of simple inconvenience is completely different from leaving out of annoyance or displeasure. We decided to use these qualitative aspects as our final guardrail. Any product spec that causes displeasure will not be reflected in the product, even if it boosts revenue.
In the final 4th experiment, the difference between Group 1 and Group 2 was the close button exposure logic (the close button is only exposed when certain conditions are met). While this exposure logic had increased both the subscription conversion rate and D1 retention on other purchase screens, D1 retention uniquely dropped in this experiment. We diagnosed the reason as "users felt displeased." That's why we decided not to choose Group 1, regardless of the calculation above.
Did users actually feel displeased? If they did, was it enough to spread negative rumors about our product after leaving? Because this final guardrail could be somewhat subjective, it required extensive discussions with various related departments. After thorough deliberation, we reached the conclusion above and felt proud that we had established our own unprecedented standards. Of course, those standards aren't absolute. They can change at any time depending on the product's situation in the future. The important thing is that when there's such a clash of values, we need to be able to gather our opinions together and figure out which direction to go and how to get there.
As a result, after much struggle, we found a balance point that satisfies both the users and us as providers. Also, with each round of experiments, our insights and talking points about users became richer, leading to significant learning and growth at the team level.
Moving forward, based on our own solid growth philosophy,
we will continue to advance healthy monetization strategies as long as they don't cause user displeasure.
And that concludes the story of the first subscription prompt experiment, spanning four rounds since the end of Q2 2023!
The Importance of the First Subscription Prompt — The secret of the onboarding purchase screen
The Importance of the First Subscription Prompt Considering Retention — Optimizing the timing of the onboarding purchase screen
The Subscription and Retention Balance Game — 2x improvement in subscription conversion vs 10% improvement in D1 retention (This post)