Growth Team's Metric Meetings (Part 2)

How Metric Monitoring Drove Improvement in Alarmy's Subscription Metrics

The most powerful motivation to keep holding metric meetings
is probably the experience of using a seed found during a metric meeting to actually drive metric improvement.

Previous post: Growth Team's Metric Meetings (Part 1) — Monitoring Alarmy's Subscription Metrics with the Growth Hacking Methodology

In this post, let's look at a case where DelightRoom's subscription squad used a seed discovered in a metric meeting to drive a significant improvement in actual metrics. As it happens, this particular case involved communication over our team messenger, so I can include screenshots of the actual conversations from that time.

[1/4] Spotting the Phenomenon and Clarifying the Issue

While preparing for our metric meeting as usual, we noticed a significant drop in our free trial conversion rate (Trial CVR). Because the metric had been trending steadily upward thanks to accumulated wins from test groups over the fourth quarter, this drop was extremely baffling. It seemed obvious that some specific change in the product—one I wasn't aware of—was impacting our core metric.

By breaking down our core metric, the free trial conversion rate, by entry point, we pinpointed the issue to a conversion drop at a specific entry point. Furthermore, when we divided that entry into a funnel of preceding and succeeding screens, we found that the entry rate to that specific screen had plummeted. An Android engineer on the subscription squad then directly checked and shared the task history deployed around that time. (See image below)

[2/4] Discovering the Seed and Syncing Direction Across Teams

Since there were no deployments from our squad at that time, we had to mobilize the collective intelligence of the entire product team, which led us to a seed: a specific task from the design group might have been the variable. Of course, because this wasn't a clear-cut A/B test, we couldn't be 100% certain that this task was the root cause of the metric drop. Plus, since that work was ultimately done to achieve the design group's goals, simply rolling it back just to serve the subscription squad's goals didn't seem like the right solution.

So, based on the seed we found, we first formed a hypothesis from the subscription squad's perspective and then worked with the design group to refine it so that it would satisfy their goals as well. Because we had a fairly high level of confidence in this hypothesis, we decided not to bother running the revision as an A/B test; instead, we deployed it to all users right away and agreed to simply do a before-and-after comparison to see if the metric improved.

[3/4] Monitoring Metrics Post-Deployment

The schedule for this task got pushed back a bit early in the year due to other major, heavy tasks, so it wasn't finally deployed until early March. Usually, I'm itching to see experiment results quickly even during a standard A/B test, but I was especially anxious for this deployment. Did our metric meeting really uncover another impactful seed? I was incredibly curious myself.

Sure enough, our secondary metric—the screen entry rate—clearly improved after this work.

Then what about the free trial conversion rate originating from that screen?

The free trial conversion rate that had sharply declined at the Wake Up Check entry point completely recovered after the deployment!

The metrics for Alarm Power-up also showed almost exactly the same trend, which firmly proved that our hypothesis was indeed a fact.

[4/4] Retrospective

It was thrilling enough just to see our core metric improve. However, I was even happier that we uncovered a seed capable of generating even more impactful hypotheses down the line. (While I can't reveal exactly what that seed was) it was a hypothesis that could be directly applied to other entry points, allowing us to aim for core metric improvements across various entry surfaces.

The subscription squad didn't just sit there helplessly suffering from the sudden drop in the free trial conversion rate at the end of the year. Thanks to our metric monitoring, we caught it right on time and quickly found the cause with very little input. We were also able to not only restore the metric but also secure a hypothesis that could deliver further impact.

Recently, in early April, while monitoring metrics, we noticed that the free trial conversion rate in the Squat mission and Step mission had noticeably jumped. Similarly, breaking down the funnel before and after showed that while the entry rate to the mission setup screen remained unchanged, the conversion rate following entry consistently increased. What's peculiar is that while there was no change in other countries, it went up exclusively in the US.

I wonder what sweet hypothesis the subscription squad will gain from witnessing this phenomenon. (Heart pounding)

I really hope that other growth organizations (teams) get to feel this same thrill when preparing for their weekly metric meetings.

And that wraps up the two-part series on the Growth Team's Metric Meetings!