How on Earth Should We Approach Increasing Retention?

Here is a thinking framework for finding ways to increase Alarmy's retention.

🙌 Hello, You're Back!

The Moment I Feel True Authenticity

When you sleep in late on a weekend and open a delivery app because you're starving, like everyone else, you look for highly-rated restaurants first. But if you just blindly trust the rating and place an order, you might be setting yourself up for disappointment. You need to verify if the reviews behind those ratings are legit. There are a few key phrases that instantly make me feel a sense of true authenticity, and the first one is "I ordered from here again." Sure, you might try a place once, but ordering again? That makes me trust the place, figuring there must be a good reason—maybe the food is amazing or the service is super friendly. Ultimately, I always find myself ordering from places with tons of repeat-order reviews.

If we apply this concept of repeat restaurant customers to an app service like Alarmy, they are our Retained Users. Depending on the nature of the service, there might be specific goals regarding what actions we want them to take upon returning, but generally, opening the app or triggering at least one active event is considered "returning." After all, they have to open the app first to do anything else.

Retention Rate (hereafter just "retention") is an absolutely critical metric for the maintenance and growth of an app service. If you run an app, you've probably heard this a million times. If you ask, "Why is retention so important?", I would answer, "Because you can maximize the net value obtained from a single customer." There are plenty of other ways to say it, but the gist is that you can reduce customer acquisition costs and maximize both the tangible and intangible value you get from one user. (This is usually phrased as maximizing a customer's LTV, or Life Time Value.) Going back to the delivery analogy, it's the exact same logic as one customer ordering multiple times, thereby increasing the revenue per customer.

So, What's Alarmy's Retention Like?

At Alarmy, we define retention as the percentage of new users on a specific day who return N days later. Depending on this N value, we call it D(N) 리텐션. We sometimes segment this by the date users first entered, or we check the average value over a specific period. (Keep in mind that the method of measuring retention can vary slightly from service to service.)

Currently, the Alarmy app's D1 retention rate is around 50%, fluctuating slightly within the 50–60% range on Android and 45–55% on iOS. This is quite high compared to other apps. 😎

Just to give you an idea, if we look at AppsFlyer's statistics, the overall D1 retention for apps is around 25%. The gap widens as time goes on; Alarmy's D7 retention sits around 40%, whereas typical apps are reportedly under 10%.

More & More

When comparing against other apps, I do feel a swell of pride thinking, 'Wow, our Alarmy is a great product!' but we can't just be satisfied and settle here. We want even higher retention! (If everyone else scores a 10 on a 100-point test and you get a 50, you can't grow if you get cocky about it.)

Because of this, in Q4 of 2021, we adjusted the goal of the Product Quality (PQ) squad—which had previously focused on improving usability—to retention, and sprinted for a whole quarter. Ultimately, the PQ squad aimed to elevate the product's quality, and since our previous target metric of usability ultimately leads to retention anyway, our goal simply became sharper.

That was a long intro. In this post, as the PM of the PQ squad for that quarter, I want to:
1. Introduce how we approached increasing retention, and
2. Share the lessons we learned through that process.

💭 Organizing the Branching Thoughts

The More You Look, the More Complex It Gets

Unlike its simple formula, retention turned out to be a truly complex metric the more I dug into it. The directions of our brainstorming and discussions were so diverse that it often felt out of control. To give you some examples of those thoughts:

• Does it only matter if they return "on" the Nth day? Isn't it better if they keep returning consecutively? Which one is right?

• Between retaining users who are leaving and getting already-retained users to stay longer, which comes first? Are these two the same thing?

• If we add new missions used for dismissing alarms, won't users stick around more?

• If the alarm setup process is easier, won't they use it longer?

As these thoughts expanded infinitely and spawned countless additional ideas within them, finding a highly impactful method became difficult. Even if a plausible idea emerged, I felt anxious, thinking, "There's got to be a more impactful task out there."

So, after a few rounds of ideation, I took some time to structurally organize our thoughts. I figured we needed to categorize the available methods to better build consensus with the squad and the company on a directional level, rather than just discussing granular tasks. More constructive discussions happen when you can clearly see where your current thought originated and what branches it could extend into.

As a result, we divided our approach into two main axes: data and intuition, and within data, we broke it down further into snapshots of a specific period vs. changes over time.

Data and Intuition

At first, these two seemed like opposing concepts, but I eventually concluded that they are relative concepts. In other words, rather than a strict binary, they exist as axes at opposite ends of a spectrum.

Ideas that can be fully backed by data leave relatively little room for intuition. Conversely, when there is absolutely zero data to lean on, we use intuition as the basis for our judgment. Most ideas land somewhere in between.

The split between data and intuition below should be understood as the "starting point of our thinking." Exactly where an idea—sparked by one or the other—lands on the data-to-intuition spectrum won't be known until we reach the end of fleshing out the hypothesis.

A. Starting from Data

1) 📸 Observing a Snapshot of a Specific Period - If 5 out of 10 stayed, why did they stay and why did the others leave?

The first approach is taking a snapshot of a specific date or period to separate the users who stayed from those who left, and then figuring out the reasons for both. The hypothesis is that if we identify those reasons and either reinforce or mitigate them, more users will stick around. You can think of it as the diachronic approach out of the diachronic/synchronic concepts we learned studying for the CSAT (Korea's university entrance exam). (Does that sound too nerdy?)

There are two main perspectives we can glean from this snapshot.

• Why did the users who stayed with Alarmy choose to stay?
  → Hypothesis: If we make the leaving users feel the same reasons that made the retained users stay, wouldn't more users stick around?

• Why did the leaving users leave?
  → Hypothesis: If we eliminate or mitigate the reasons they left, they won't leave.

To put it differently, users returning to the app means "not leaving"—preventing them from deleting the app or never opening it again. This is probably why Churn rate is used as the opposing metric that goes hand-in-hand with retention.

Making them stay or stopping them from leaving—aren't they exactly the same thing? Are you just playing with words?! You might think that, but if you actually list out the methods derived from each perspective, you'll see they are different. In other words, returning and not leaving aren't perfectly equivalent, but this way of thinking lets you look at the same number or phenomenon from different angles.

Let's look at a few major questions based on each perspective.

Even after digging just one layer deeper, you can see the differences in the questions (hypotheses) derived from the two perspectives. In fact, as you flesh out each hypothesis, they evolve into entirely different problem definitions. On top of that, if you cross-reference them with characteristics by country or user segment, you get highly specific hypotheses that are ready for immediate solution brainstorming. Of course, during that process, we look at actual user data to steer the hypotheses and weed out the most impactful ones.

2) 🎥 Observing Changes Over Time - Why is this month's retention lower than last month's?

The second approach is to break down the phenomenon over the flow of time. If the first method was taking a snapshot, the second is playing it like a video and observing where the fluctuations occur.

You lay out the retention graph over time and analyze the differences between the low points and high points. Visually, it looks like the image below. How long of a data period you observe—whether it's a year, a month, or a day—is entirely up to you. At the deepest level, you can examine user flows screen by screen to see their behavior within that specific funnel.

Let's look at some example ideas.

• During the day, when is it used the most? It peaks at 6 AM, 1 PM, and 8 PM.
  → What are they coming in for during each of these time slots?

• During the week, which days is it used most? It peaks on Monday, drops slightly by Friday, and sees a massive drop in usage over the weekend.
  → Why does usage drop between Tuesday and Friday?
  → Why does usage plummet over the weekend?

• Throughout a month, a quarter, or a year, when is it used the most?

This approach ultimately requires bringing in the first method. To define concrete problems and derive solutions, you have to take snapshots of any observed anomalies and run a comparative analysis on user behavior within those points.

B. Starting from Product Intuition

If A started with data, this approach starts by generating ideas using the PM's intuition and then reverse-engineering the evidence. The two biggest sources for honing this product intuition were user suggestions through VoC (Voice of Customer) and high-confidence ideas from within the team. A few examples look like this:

VoC Seeds

1. 미션을 해제할 수 없는 경우들이 있는데 이 때 곤란해서 앱을 삭제하게 된다. 비상 시에 사용할 수 있는 탈출구를 만들어달라.
   → What are the scenarios where missions can't be dismissed? (Threw away the barcode, moved far away from the photo location, etc.)
   → What pain points do users feel in each scenario? How can Alarmy solve this for them?

2. 아침에 더 자고 싶어서 기기를 끄거나 앱을 삭제한다. 기기를 끄거나 앱을 삭제하지 않게 해달라.
   → Why wouldn't they want to dismiss the Alarmy alarm in the morning?
   → They could just dismiss the alarm, so why turn off the device instead?

Team Confidence

1. If we properly explain core features to first-time users, won't they use the app longer?
   → What features are early users currently using? What questions might they have?
   → Which features should we introduce to help them use the app better?

2. If we give positive feedback after their initial use, won't they use the app longer?

Beyond these, ideas flow from a variety of sources, like referencing other services. As mentioned earlier, even if these ideas begin with intuition, you can find the data to back them up from both inside and outside the service.

As Always, Lessons Etched in My Mind

My current conclusion on how to increase retention is: "There is no single right answer. You have to approach it from multiple angles and communicate with users." After such agonizingly long thoughts, what a seemingly obvious conclusion...! Fortunately, though, all that wandering around helped me formalize my own starting framework for finding ways to boost retention. Let's wrap things up by summarizing the lessons I picked up along the way.

1. ⚖️ Data is important, but intuition is too. You always have to balance them well.
   • Data is a great tool for persuading others, but you can easily fall into the trap of trying to prove everything before you even start. If you obsess strictly over data, you might get just as bogged down as when you make decisions based solely on intuition.
   • Just like seasoning soup, constantly aim for that "just right" spot and always check that you aren't leaning too heavily to one side.

2. 🏃 It's better if users have a variety of reasons to return.
   • In Alarmy, the motive for returning is setting or dismissing an alarm. If we had other funnels, we could generate far more impactful hypotheses.
   → This is something I'm really looking forward to, as core in-app features are expanding rapidly according to Alarmy's product roadmap. 😎

3. 🤝 At the end of the day, retention is just a metric. You need alignment on how to approach it.
   • Just as the retention formula varies by service, the direction of how to approach it also varies. Reaching a consensus on this is key.
   ; Based on Alarmy's current formula, we could theoretically boost retention simply by spamming push notifications to artificially increase the number of users logging in on a "certain day." The metric might go up, but we'd need to pause and agree internally on whether that's actually what we want in the long run. (Luckily, the Alarmy team was already aligned on our long-term direction.)
   • As you work to increase retention, defining leading metrics can help sharpen your problem definition. Because retention is such a lagging metric, we had an overwhelming amount of ideas flying around early on. It's a good idea to prune those branches through convergence after the initial ideation phase and seek out leading metrics instead.

The methods in this post won't account for every possible edge case, nor will they magically pop out a clear-cut solution. But as the PQ PM, this process helped me establish a visual framework for which direction to look at the problem from while defining problems to boost retention.

I'll wrap up here, hoping this provides at least a little help to anyone wrestling with retention.