Growth Team's Metric Meeting (1)

Alarmy's Subscription Metric Monitoring Based on Growth Hacking Methodology

"Growth hacking" is a product growth methodology that builds meaningful lessons around improving core metrics through iterative hypothesis-testing experiments. It's a continuous, repeating process: planning hypotheses to improve metrics → prioritizing them by calculating impact, confidence, and effort → deploying the experiment through actual development work → analyzing the results to build lessons and stack up follow-up hypotheses → and so on.

While other engineering organizations might already have similar growth hacking collaboration processes in place, they often neglect metric monitoring, which is a crucial step in that process. The main reasons are likely tight resource environments where teams have to immediately look at the next backlog item without taking the time to properly measure the impact of their latest deployment, as well as organizational cultures that receive top-down backlog planning seeds from partners, business units, or executives rather than discovering them from data. Although collaboration processes naturally get tuned to fit each organization, if you're going to attach the term "growth hacking," metric monitoring is a collaboration process the team absolutely must prioritize.

What is a Growth Team's Metric Meeting According to Hacking Growth?

According to Sean Ellis's Hacking Growth, the best growth hacking teams go to great lengths to ensure their data collection and analysis are proper and precise. In fact, Facebook's growth hacking team took the extreme measure of halting all engineering experiments in January 2009 to dedicate the entire month solely to advancing their data tracking and collection capabilities. That's how critical it is to collect and manage the metrics that define product growth. Ellis also noted that all members of a growth team—including engineers and designers—should collectively monitor the trends of core team metrics and proactively voice their opinions on directions for metric improvement. This is because a growth organization isn't just a simple hodgepodge of individual functional teams, but a mission-driven organization specially formed to improve specific metrics.

DelightRoom Subs Squad's Metric Meeting

The 'Subs Squad' is the growth organization responsible for boosting subscription revenue for DelightRoom's Alarmy service, and we treat the metric monitoring process with high importance.

There are two purposes for the Subs Squad's metric meeting. The first is to check for any major issues in our subscription metrics, and the second is to unearth seeds for improvement. If external variables played a huge role in the fluctuations of our metrics, it would be tough to find meaningful seeds. However, due to the nature of Alarmy's subscription revenue—which is generated purely by the power of the product—internal variables have a bigger impact than external ones, making it entirely possible to dig up meaningful seeds. When we determine that a fluctuation is caused by an external variable, we often just skip any further analysis.

<Metrics we check for potential issues>

These are the 4 core metrics directly related to subscription revenue.

[Reference: Understanding the Basic Metrics of Subscription Monetization (1) ]

Trial Conversion Rate (Trial CVR)
Trial to Paid Conversion Rate (T2P CVR)
Cohorted Churn rate
Monthly Recurring Revenue (MRR)

<Metrics we check to uncover seeds for improvement>

For this, we need to look into detailed metrics that influence the core metrics. Examples include the 'trial CVR by entry point', the 'funnel drop-off rate up to key entry points', and the 'funnel drop-off rate within the purchase screen'. Depending on the situation, we also check custom metrics as needed.

(The data swamp has a fatal attraction; if you look too deeply, two hours can easily fly by, and you sometimes find yourself looking at largely irrelevant data. So, you need the decisiveness to cut it off at a reasonable point and surface back up.)

Every Thursday morning, all teammates (iOS Engineer, Android Engineer, Product Designer, Product Manager) gather to review the previous week's metric data for anywhere from 30 to 50 minutes. The PM prepares the monitoring notes in advance. Aligning with the purposes mentioned above, they check for anomalies in core metrics and neatly organize any additional data they looked into to unearth related seeds. During the metric meeting, members look over the compiled notes, propose various hypotheses that might have influenced metric changes, and exchange feedback. "It looks like there was an overall improvement of about 20% using the winning experiment group as a baseline." "The last deployment included these tasks from another group, could that have caused cognitive dissonance?" "I think engagement dropped because of the Thanksgiving holiday period in the US." Even if a hypothesis sounds a bit absurd, we share it comfortably and freely. Thanks to this premise of floating free thoughts, the Subs Squad's metric meetings are always highly immersive and consistently yield a colorful array of hypotheses.

Lastly, separate from achieving those two main goals, the metric meeting keeps teammates aligned with a single, shared sense of purpose. A metric meeting isn't just a foundational process for finding product growth hypotheses; it's also a process that lays the groundwork for creating true team synergy.

At first glance, a metric meeting is hard work and quite a hassle. Since it's not easy to pinpoint sharp seeds compared to the considerable time spent looking at the same data over and over, it's easy to feel like it's just a meaningless, formal chore. If any growth team has fallen into this kind of rut, I recommend checking the following three points.

1. [Relevant Data] Are you looking at data directly tied to core metrics?

- To find sharp seeds, you must properly define the secondary metrics that heavily influence your core metrics.

2. [Monitoring Efficiency] Is your monitoring process efficient?

- If there's a metric processing workflow you go through often, including frequently checked secondary metrics, you should automate the repetitive parts to minimize monitoring effort.

3. [Team Engagement] Are all teammates fully engaged together?

- Discovering seeds requires proposing ideas from multiple angles. One member might tend to look at metrics based on the funnel, while another might interpret them based on user attributes. Also, from the perspective of the person preparing the metric meeting, they can sometimes get too buried in the data during the analysis process, making it hard to take a macro view. Teammates need to engage together and provide input so no one gets trapped in a biased perspective or narrow tunnel vision.

But more than those three points, the most powerful motivation to keep metric meetings going is...

...actually experiencing metric improvements driven by the seeds you discovered during those meetings.

In the very next post, let's look at an example of how DelightRoom's Subs Squad massively drove actual metric improvements using a seed we unearthed through a metric meeting!

mail : stephan@delightroom.com
instagram : @seosinghan