How We Succeeded With Our Onboarding Experiment After a Year

How We Boosted Day 1 Retention by 20%

1

Retention is a wildly important metric here at Alarmy, and we've tried countless approaches to improve it. Among all the experiments we ran over the past year, I want to share the planning process and lessons learned from our post-install app onboarding experiment, which drove the biggest bump in retention.

Which Stage of Retention Should We Focus On?

Saying "retention improved by x%" can be quite tricky to interpret. There are various stages like Day 1, Day 7, and Day 30, and each carries a different meaning.

Ultimately, you need to specify the exact timeframe—like "Day 1 retention improved by x%" —to truly understand the impact. The further out the timeframe, the bigger the impact of improving retention, but the harder it is to actually achieve.

So, which retention point should we focus on? For most products, the biggest drop-off happens within the first 24 hours right after installation (= Day 1 retention).

Alarmy is no exception. Whenever we managed to improve Day 1 retention, the entire curve shifted upwards, often leading to improvements in Day 7 and Day 30 retention as well.

How Can We Make Users Experience the "Wow Moment" Better?

For Alarmy, the wow moment is when the alarm actually goes off, which usually happens the morning after installation. But when we discovered that a staggering 40% of users churned before even experiencing their first alarm the next morning, we started asking ourselves this question:

How can we get as many people as possible to experience the alarm (the wow moment) the very next morning?

Looking at the data, it was (unsurprisingly) clear that users who set an alarm on their first day were much more likely to stick around and experience the alarm the next morning. Getting as many people as possible to set an alarm on day one was absolutely crucial.

In the end, we figured that if we nudge users to set an alarm during the onboarding process right after installation—when their interest is at its peak—the chances of them experiencing the alarm the next day would go up. This led to the following hypothesis:

If users configure an alarm exactly how they want it during post-install onboarding, the percentage of users experiencing an alarm the next day will increase.

Alarmy's Revamped Onboarding

We designed an onboarding flow to help users instantly grasp Alarmy's value, think "I want to use this!", and immediately set up an alarm right after installing.

1) Briefly and clearly express the value our app provides.

2) Walk users through the process of setting an alarm with their preferred settings.

→ The Final Onboarding Screens

Experiment Results

• Day 1 retention jumped by 15% to 20% with the new onboarding compared to the old one. Even better, this trend held steady all the way to D7 and D14.

• There was some concern that making the onboarding process longer and removing the "skip" option might cause a higher drop-off rate during onboarding itself. However, the upside from the new onboarding far outweighed the number of users dropping off during the flow, ultimately leading to an overall retention boost.

Key takeaways

Wrapping up the lessons we learned through this process:

• Figure out the core wow moment your product needs to drive.

• Identify what your product's wow moment is and map out the user's journey to get there.

• Brainstorm ways to get users to experience that wow moment as quickly as possible post-install.

• It's highly effective to push for the wow moment when users are most interested in using the app (typically during onboarding right after installation).