Exploring Alarmy User Patterns Through Machine Learning

How are our users actually using the app?

Shortly after joining DelightRoom, I read a post analyzing DAU, and while chatting with the team, a question popped up. Could we go beyond knowing how much users use our service and figure out exactly how they use it? Alarmy's mission is to wake users up successfully, but how are our users actually waking up? And does the way they wake up affect our retention rate?

Understanding how users interact with the app is a topic that anyone working on a product—not just data analysts—wants to know and think about at some point. While everyone has a mental image of the user based on past interviews and empirical data, I want to briefly share my experience of building a more holistic view using actual user data. Of course, there's no single right answer here, but I believe it serves as a great starting point.

Where do usage patterns diverge?

Before analyzing user usage patterns, the obvious prerequisite is defining the components of those patterns. I started by thinking about which factors could define user patterns, essentially tackling what we call Feature Engineering.

I started with the basics tied to app features: whether individual alarms had missions attached, the number of missions, and the snooze count when an alarm rings. But considering the specific domain of waking up, I also uncovered and defined additional user-level features beyond just individual alarm settings—things like the total number of alarms, total wake-up duration, the interval between alarms, and the average time of the alarms.

Also, since this is a service used every day, I wanted to eliminate the cyclical data noise between weekdays and weekends. I separated the defined features into weekday and weekend buckets and designed it to use period averages to suppress outliers as much as possible.

The classics are classics for a reason

Figuring out how to segment these well-prepared user features was another point of deliberation. I could have simply looked at the distribution per feature to understand what values users held for each factor, or I could have used deep learning or prediction models to isolate features that drive high retention and segment users that way. While many approaches came to mind, the method I ultimately chose was clustering via KMeans.

I wanted to look at overall usage patterns without any bias from predefined labels, and I thought it was the most fitting approach to see if we could segment user patterns based on a combination of all factors rather than just a single element. Plus, in an era where LLMs are stealing the spotlight, a part of me wanted to prove to myself once again that traditional models are still incredibly powerful. LLMs are great, but Machine Learning isn't dead yet...!

KMeans Clustering

For the algorithm, I used Python Scikit-Learn's KMeans, and for selecting the number of clusters, I used the classic elbow method utilizing inertia.

Honestly, selecting the number of clusters using the Elbow Method inherently requires some personal judgment. I always follow what Andrew Ng said in his Machine Learning course: "Choose a number of clusters that you or your team can actually handle." This means that while you should reference inertia, even if 10 or more clusters show low inertia, it's better to reduce the number if a single person would struggle to understand and manage all 10. Keeping this in mind, I also limited the number to 4 clusters to train the model, aiming to keep my own understanding clear and to facilitate internal communication.

Adding a dash of packaging to Machine Learning

In a recent podcast, the CEO of Perplexity noted, "In the end, every service is just a wrapper for another service." The point was that since OpenAI is a wrapper for Nvidia, and Nvidia is a wrapper for semiconductor chips, the Application Layer that creates the real value is what's truly important. Similarly, the clustering results obtained from KMeans ultimately change in value depending on the interpretation and insights attached to them.

If you use ._centers_ on a model holding KMeans results, you can check the centroid values of each cluster, and for the data analyst or scientist who built the features and thought deeply about the model, that alone might provide plenty of insight. However, if there are other stakeholders involved, simply presenting numbers can easily become just another set of floating digits.

One method is to give each cluster a name that intuitively represents your understanding (much like defining user personas). It might feel a bit embarrassing or cheesy, but since most people digest text and use their imagination far more easily than they do with raw numbers, it's undeniably a more effective way to communicate your findings.

I can't claim it's a perfect example, but it's undoubtedly far more intuitive than simply saying, "Cluster 0 has an average of 2 alarms, a 5-minute interval, and takes about 5 minutes to dismiss~"

Taking it one step further

Now that we've defined user features and applied persona-like names through meaningful clustering, the remaining possibilities are endless. We can check the status of key metrics for each cluster, define which user patterns are currently contributing more to our service, and even map out a natural user journey by observing metric shifts as users transition between clusters over time.

For Alarmy, retention is a metric directly tied to DAU. By checking the retention over time for each cluster, looking into detailed mission usage patterns per cluster, and tracking retention shifts based on state changes, we were able to extract even more insights.

For additional analysis, I recommend saving the model itself as a file like .pkl to maintain model consistency using load(), and separately storing each user's features and cluster labels.

Finding the right approach...!

As a data practitioner who boosts productivity with the help of various AI tools like ChatGPT, Claude, and Cursor in the field, the breakthroughs provided by LLMs are incredibly exciting and fun. Whether it's the near or distant future, this project really made me realize that there are still plenty of areas where traditional Machine Learning or statistical analysis—rather than just LLMs—remain highly relevant and meaningful. (Though, of course, when writing the code, I absolutely worked alongside Claude.)

If you're wrestling with similar questions, I highly recommend giving it a try!