Alarmy Launched a 'Sleep-Inducing' Feature in Just 3 Months

What We Would Have Done Differently If We Could Turn Back Time

1

Previous post: After 9 Years of Waking You Up, Alarmy Launched a 'Sleep-Inducing' Feature in Just 3 Months
Sleep Sounds Launch (Part 1): How did we launch so fast?

In our previous post, we looked at how the Sleep Sounds feature was launched in just three months.

We narrowed down our target users, defined the core problems they were facing, and streamlined our product spec to focus solely on solving those specific issues. We also reduced time by adequately outsourcing human and material resources when starting the work, and solidified detailed user stories through a "seeing is believing" process (discussions based on visual materials, refining specs while testing quick builds, etc.). That's how we were able to quickly present our sleep feature to our users.

As anticipated at the end of the previous post, looking at user reactions and data brought up several points for a retrospective on our journey. If I could turn back time, what would I have paid closer attention to?

Sleep Sounds Launch (Part 2): What should we have watched out for when launching so quickly?

1. Set a target "core metric," but also establish a closer "initial leading metric." Consider making the leading metric the MVP goal.

The purpose of launching Sleep Sounds was to shorten our users' sleep onset latency. If we expressed this as a metric, we would measure how much the time it took them to actually fall asleep decreased. However, since this is physically impossible (for now), we had to set up proxy metrics instead. There were several candidates to represent this, such as "setting the sleep mode we provide after playback" or "maintaining sound playback itself for a certain period of time." But these "core metrics" are generally lagging indicators and tend to be hard to achieve in the short term.

Fundamentally, because new features are planned under a macroscopic product roadmap, their target values tend to be somewhat ambitious and visionary. It's inevitable when you start out grandly with thoughts like, "I'm going to solve users' sleep problems with Sleep Sounds!" Because of this, core metrics struggle to act as sharp guideposts for solving the problems you face right after launch when the user base is still small. Was it uncomfortable while falling asleep, did they not even attempt to fall asleep, did they experience discomfort while choosing content, or was the problem that they didn't even enter the sleep tab... It's difficult to judge where we should focus our efforts right after launch using just a single core metric.

From the user's perspective, a new feature is also very unfamiliar. Will users who installed our app expecting an "alarm app" be interested when they see a "sleep" feature? Even if they are interested, it's not easy to quickly understand the various custom functions we provide, including "sleep mode." We need to work simultaneously on aligning their expectations with ours while gradually raising their level of understanding of the provided features. Considering all this, achieving an ambitious core metric in the short term is very difficult.

So, we decided on a very simple initial leading metric. We set our front-end goal as "getting users to play just one of our Sleep Sounds contents." More specifically, we used the ratio of Sleep Sounds playbacks to app opens among users on their 7th day since installing the app as our leading metric, and we reverse-calculated the target level from the core metric.

"Achieve an n% playback rate for 7th-day users!"

Thanks to this, the direction of our product spec narrowed significantly, and it became clear where we needed to focus our efforts.

On the other hand, I felt a pang of regret wondering what it would have been like if we had focused even more on this leading metric when launching the MVP from the start. What if, instead of deploying a version that covered the entire journey up to the core metric, we had polished just the early part of the journey, released it first, and then iterated based on metrics and user feedback? Both approaches have pros and cons, but it was dizzying to realize I hadn't even considered this method at the time. I guess I was being too ambitious as well. Haha.

2. Set specific target values for hypotheses formulated before launch so they can be properly validated afterward

As is always the case with "first-time" feature launches, there is no data to back up the reasons behind the product spec. We base things on minimal user interviews, research materials, and competitor references, but these only serve to provide direction in a broad context. A surprisingly large part of what determines the detailed specs of a newly launched feature is intuition about the product, and since this varies quite a bit from person to person, reaching a consensus is not easy. When opinions are dramatically narrowed down, we're busy putting them straight into action. The refreshing feeling of seeming to have solved a difficult problem, coupled with the tight squeeze of limited time, are the reasons. We didn't leave a record of "why" we made those product specs, or "what data" we would use to cross-check them post-launch.

After the feature launched and we faced the data we had been waiting for, we had no idea where to start making improvements. Just like when we "first" launched, it felt like we still lacked data to back up the reasons behind our product specs. Come to think of it, it was the right time to check if our initial hypotheses actually hit the mark. So, we reorganized the rationale and hypotheses behind the various specs included in the release version.

The reason we placed the Featured section at the very top was to help users with their initial exploration. >> Are they utilizing Featured well for their first exploration?

The reason we created the "sleep mode" feature was to help users who chose Sleep Sounds content set up their sleep environment. >> Is sleep mode being used well to help them set up their sleep environment?

The reason we ordered the categories as Nature > ASMR > … > Music was because that was the order of popularity during user interviews. >> Does this actually reflect user popularity?

..

Fortunately, we were able to cross-check the above using various event values we had planted here and there. After looking into it, we could roughly see which of our initial hypotheses had issues right away. However, because we hadn't set specific target values for the initial hypotheses, and since the events weren't sharply designed specifically for this validation, we had to review things somewhat clumsily. Of course, it would have been truly dizzying if we couldn't even examine this much. No matter how busy we were, we should have set rough target metrics for each hypothesis to ensure they were working properly and designed initial events to verify them later on. If we had done that, we could have drawn up higher-quality follow-up product specs.

Finally,

3. Anticipate complaints and bugs among early user feedback, and secure resources in advance for proactive or rapid post-launch responses

Given the size of the update, the user reaction was intense. Quite unexpectedly, the main complaints were about the disappearance of the 6 to 8 Sleep Sounds contents we previously provided. The 700 or so high-quality new contents we offered were useless to them. The loss of the sounds they used to listen to was a bigger deal. Although it was a very small minority among our 2.5 million total users, the heat of their reaction felt as intense as if they'd been stripped of a daily necessity that made their lives difficult. It makes sense, considering this wasn't just simple content—it was sleep-related content they couldn't fall asleep without...

We immediately opened up a path within the product so they could access and listen to the old contents, and user complaints vanished quickly. In fact, feedback thanking us for our fast response followed. Only after the complaints demanding the return of the old content cleared up did we start seeing feedback focused on the new Sleep Sounds.

There are two regrettable things here. First, while we spent a busy three months launching the MVP, there was more than enough time to inform users in advance about the newly launched feature. If we had verified beforehand just how precious those 6 to 8 existing Sleep Sounds were, we could have launched with their access path wide open. Second, although we managed to respond in a hurry, it was a completely unforeseen issue that forced all members to suddenly pull resources away from other work. Considering it was a fairly large-scale update, if we had set aside some buffer resources specifically for handling bugs and complaints, we could have managed the situation much more smoothly.

It took three months to launch the Sleep Sounds feature, and two months have already passed since then. Initial data from an ample user base is piling up, and we are currently working on various improvements to increase the playback rate based on user feedback. The metrics we need to look at are clearer than they were during the initial launch, and we have a lot more qualitative and quantitative data to use as seeds, but we still have a long way to go. We need to quickly increase the playback rate as a leading metric and advance toward our ultimate goal of "shortening sleep onset latency." Keeping in mind these three points of regret from the launch process, I need to ensure a steadier hand on the wheel for the inevitable large-scale product spec seasons in the future.

I hope we will have grown into an awesome product that solves sleep problems for even more users by the next Q4.

End of Sleep Sounds Launch (Part 2)!

Sleep Sounds Launch (Part 1): How did we launch so fast?
Sleep Sounds Launch (Part 2): What should we have watched out for when launching so quickly?