The Importance of Prompting the First Subscription with Retention in Mind

Experimenting with the Optimal Timing to Show the Paywall During Onboarding

Following up on the previous lessons, we are currently running the next experiment below.
We've achieved a much larger growth compared to when we didn't show the onboarding paywall at all. However, realizing that changing the timing of when it appears could yield an even higher conversion rate, we are continuing our experiments to find the optimal timing that fits our product.

At the end of my previous post, The Importance of Prompting the First Subscription — The Secret of the Onboarding Paywall, I mentioned that we were running an experiment to show the onboarding paywall even earlier.

Given that the funnel drop-off rate isn't very high during onboarding, showing it earlier probably wouldn't contribute much to increasing the impression volume itself. Rather, as discussed in the previous post, we ran this experiment with strong confidence that exposing it earlier would increase the conversion rate to subscription rather than just boosting impressions.

The question was, "At what point should we show it to generate a higher subscription conversion rate without hurting new user retention?" If it were a purely premium service with no free features, we might not have to worry much about retention. But for Alarmy, which offers some features for free and generates significant ad revenue, retention is one of the most critical metrics. Since the daily volume of new users is massive, even a slight bump in D1 retention leads to a substantial increase in cumulative DAU. Furthermore, when comparing the difficulty of "increasing new user retention" versus "increasing subscription conversion rate," the former is obviously much harder. Therefore, in a showdown between a drop in retention and a boost in subscription conversion, the former is almost always going to win. Thus, finding a variant with no retention drop was crucial.

01. Core Metric — Subscription Conversion Rate

Generated a 60% ~ 80% lift in subscription conversion rate

Sure enough, the subscription conversion rate was a landslide victory for the variants that showed the paywall earlier. Although the extent of the lift varied slightly depending on the OS and country, they all shared one thing in common: the "variants with earlier exposure" showed the highest conversion rates.

The detailed metrics are as follows.

Another fun fact here is that the only flow difference between the "before permissions" variant and the control group was whether users saw the permission request dialogs before or after the paywall.

To think the conversion rate jumped 1.3 times simply because we popped up the paywall before showing the dialogs...

I was surprised that just "showing it earlier" increased the conversion rate so significantly.

So, was their D1 retention unscathed?

02. Negative Metric — D1 Retention

Experienced a 2.2% ~ 8% drop in retention

Sadly, D1 retention dropped. Ouch...

There were a few interesting anomalies here, too.

Unlike the subscription conversion rate, which grew higher the earlier the paywall was shown, the variant with the lowest retention was the "before permissions" group. In other words, that variant ranked 2nd in conversion rate but dead last in retention, while the earliest exposure variant ("right after value proposition") ranked 1st in conversion rate without being dead last in retention.

The control group and the "before permissions" variant differed only in whether the permission request dialogs appeared before or after the paywall, yet there was a gap in retention.

While a D1 retention gap existed between the control and variant groups, they tended to converge as they approached D7.

This last characteristic is where my inner turmoil began.

If the retention gap practically disappears by D7,
wouldn't it be okay to select the variant with the high subscription conversion rate as the winner?

I know D1 retention is highly valuable, but
when comparing a 2.2% drop in retention against a 50% boost in subscription conversion rate,
which is more valuable for the product as a whole?

03. Selecting the Winner

2.2% ~ 8% drop in retention vs 60% ~ 80% lift in subscription conversion rate

To calm my inner turmoil, I compared the value of a 1% lift in new user retention against a 1% lift in subscription conversion rate.

Given a daily new user volume of A, a retention lift of B% led to a DAU increase of C users, which, when multiplied by ad revenue LTV, calculated to an additional revenue of D won. Similarly, for the same new user volume A, a subscription conversion lift of E% led to an increase of F subscribers, which, when multiplied by subscription LTV, calculated to an additional revenue of G won.

Finally, I completed the equation by factoring in the difference in difficulty (the hit rate of experiments) between raising retention and raising subscription conversion. And for the very, very last touch, I sprinkled in a bit of the user's annoyance level, which is hard to reflect numerically...

The conclusion I arrived at was: Control group wins...!

04. Follow-up Hypothesis

Did the users in each variant make it to the end of the funnel?

It was pretty disappointing, but one thing was clear.

For one, pushing up the exposure timing increases the conversion rate. The other thing wasn't clear. Why did retention drop?

So, I took a closer look at the onboarding flow funnel for each variant.

And the conclusion was as follows.

"When the paywall is shown during the onboarding process, users bounce from the app right at that moment."

They abandon the app before even reaching the home screen.

Why do they do that? We need to solve this to increase the subscription conversion rate while preserving retention.

The intuitive hypothesis I came up with was,
"If the paywall is shown early in onboarding, users mistakenly think Alarmy is a strictly premium app and immediately abandon it." Thus, we are drafting mockups to prevent this kind of misunderstanding specifically for the onboarding paywall.

In a way, this is product spec added in a direction that helps users bounce from the paywall, so the lift in subscription conversion rate will probably drop slightly. However, since preserving retention is a more critical task, we decided to take a somewhat ironic approach.

Will these variants manage to survive in style...!