The Experiment That Drove 16x Growth—and Still Lost

What matters more than shipping features is the size of their impact

When asked what the most critical core competency for a PO is, I always answer without hesitation: "the ability to define problems." Finding and defining impactful problems amidst the countless user journeys within a product—I believe that's the starting point of a PO's ability to drive a product's growth. And I usually explain the reason simply: if you can't uncover impactful problems, no matter how fast you lead the team, you won't drive any product growth at all.

Even though I always gave that answer, while running full steam ahead through Q3, I ended up executing several low-impact product specs that didn't help our product's growth whatsoever. As I shared in my quarterly retrospective post (A More Special Quarter for the Subscription Growth Team), we certainly achieved growth in core metrics through various good lessons last quarter, but behind the scenes, there were also a handful of critical mistakes. I want to briefly look at just one example of those cases and reflect on how I ended up making such product decisions. Furthermore, I'll organize what action items I'll take in Q4 to avoid repeating the same mistakes.

The experiment group that drove a 16x improvement lost to the control group.

Participation in the invite-a-friend promotion was very low. I did think the incentive for inviting a friend—an "Alarmy Premium pass"—was a weak motivator, but I judged that we lacked the product planning and engineering resources to integrate other rewards (like Starbucks or Olive Young gift cards). Therefore, we decided to run an experiment nudging users who would likely be interested in Alarmy Premium to invite friends.

We defined users who dropped off after reaching the checkout screen on the Premium paywall as "users interested in Premium but unwilling to pay," and nudged them to invite friends. As a result, the conversion rate from generating an invite link to actually sharing it skyrocketed by about 16 times. However, despite that, it failed to beat the control group.

The reason was that, separate from the conversion rate increase, the absolute base size for the increment was just too small. Improving that flow generated an increment of about 50 to 80 "share invite link" actions per month, which is a minuscule increment when considering the expected value of actual user acquisition. From the start, the base number of users reaching the checkout screen on the Premium paywall itself was small, so a 16x increase within that base didn't significantly impact the overall picture. In short, it was an experiment where I completely failed to calculate the expected impact properly.

Why did I make this mistake?

1) An obsession with limited resources — The team's growth

For a long time, our organization had struggled with a shortage of engineering resources. There was a long period when it was hard just to ship planned specs on time. Thus, our main strategy was often to reuse existing code (specs) as much as possible and aim for big impact with simple product decisions. We had actually been creating solid growth with those kinds of specs. Consequently, I gradually put more focus on drafting specs lightly, and I started running mainly experiments that we could reliably win. Of the ICE (Impact, Confidence, Ease) framework, I had been focusing heavily on Confidence and Ease. But our current team is no longer like that. We have more engineering teammates than before, and their skills are outstanding. We are now in a situation where it is perfectly fine to place a much heavier weight on Impact.

2) The depletion of low-hanging fruit — The product's growth

As I just mentioned, the reason we could create solid growth even under resource constraints was precisely because there was so much low-hanging fruit. Because engineering resources were scarce, we had quite a backlog of specs that were high in all three: Impact, Confidence, and Ease. Various experiments in the new user onboarding process, various experiments on the paywall, multiple experiments utilizing price and subscription duration, etc.—while churning through around 15 experiments every quarter, we squeezed out every last drop of value we could. Over several quarters, the marginal utility of our growth gradually decreased. It became harder to maintain a high experiment win rate, and it became difficult to secure large improvements. Now, the time has come to move on to high-hanging fruit. In other words, the product has grown to a point where it's hard to expect meaningful Impact easily within specs that just have high Confidence and Ease.

Looking back, the team had grown and the product had grown, but I was the only one still stuck in the past. Haha.

Back to basics

Finding and defining impactful problems amidst the countless user journeys within a product—I believe that's the starting point of a PO's ability to drive a product's growth.

Even if a spec is a bit risky and has some engineering difficulty, our current team and product have grown enough to handle it. And for steeper growth going forward, it's time to truly focus more on Impact.

I plan to pursue a leap forward in Q4 with the following action items.

1) Estimating Impact Size

For each spec, I will forecast the expected impact size in advance. The goal isn't necessarily to predict it with absolute accuracy, but rather to build a habit of caring about the impact of the specs, even if it's just a rough estimate. Furthermore, during the process of calculating impact, I also expect that the spec might be adjusted in a direction that can maximize that impact even further.

2) Obsessing more over the "Why"

3) Talking together frequently

I will set aside time to talk more frequently and deeply with my teammates about Impact Size and the "Why." Through this, I want to naturally embed into our team culture that solving an impactful problem is more important than just increasing the task completion rate.

What was harder to bear than the final outcome of having a negligible impact on the product was the fact that, without even realizing it, I hadn't been thinking deeply enough about impact. It's true that the countless specs I wrote in the past ultimately led to solid growth, but a sense of regret follows: if I had obsessed over impact more, wouldn't it have led to even greater growth? Also, the fact that, due to a PO's oversight, the opportunity for my teammates' resources to be used more impactfully was lost is quite a painful reality.

Now, in step with the growth of the team and the product, it's time for me to grow as well.

Let's burn bright this Q4 and finish 2023 with a massive impact.