What happened when we focused on impact

Expected changes, and even unexpected positive ones

The core message of my previous post — The experiment that drove 16x growth but still lost — was that "the magnitude of the impact matters more than just shipping a feature." So, as mentioned in that post, starting this quarter we decided that every product spec must include the estimated impact size at a level everyone can understand. Even for the most minor feature updates, we agreed to 'unconditionally' practice writing them in the same format at first. Once these impact discussions become a habit and settle into our culture, we can afford to be more flexible.

We are now just past the halfway mark of the quarter, and looking back, the effect of these impact discussions has already been immense.

#1. The Expected Changes

A. "The impact is smaller than I thought..." Facing reality

After calculating the impact as shown above, most cases ended up with the realization: "The impact is smaller than I thought..." This makes sense because writing a product spec usually starts with a vague gut feeling of "Oh, this might work." Naturally, inflated hopes get popped by the calculator more often than not. (After all, rarely does anyone start writing a spec thinking, "Is this even going to work?") Brilliant ideas that popped into my head while using other products, digging through user VoCs, or looking at various data, would mercilessly shrink once they were put on the chopping block of impact calculation. I learned to squarely face the bitter reality upfront.

B. Why the impact was smaller than expected: correcting misunderstandings about target users

The reason impact was usually calculated to be smaller than expected was often due to a lack of proper understanding of the target user. Let's take a really simple example: a hypothesis stating, "If problem A on the purchase screen is solved, the subscription conversion rate will increase 1.5x." Here, we often make the mistake of thinking the target users are "users who visit the purchase screen." But actually, it's accurately "users who experience problem A when visiting the purchase screen." Because of this, the impact needs to be recalculated. If the proportion of users experiencing problem A is about 30% of the users visiting the purchase screen, the expected subscription conversion rate increment is 0.7*1.0 (users not experiencing the problem) + 0.3*1.5 (users experiencing the problem * improvement rate) = 1.15x, which is significantly smaller than the initial calculation.

C. More time spent on product specs, leading to deeper and more meticulous thinking

It takes more time than before just to define the problem and estimate the user base experiencing that problem. I check rough data and, if necessary, refer to the weekly number of VoC tickets. Also, estimating the expected conversion rate increase takes far more time than anticipated. I end up looking at how the leading metrics that precede the target conversion value will change, and I pre-examine what the existing conversion rate is when narrowing down the segment to the target users, as much as possible. Actually, taking this meticulous approach is the right way from the start, but it makes me wonder if I had been writing product specs too hastily all along under the guise of "flexibility and efficiency."

Spending a long time on planning and extracting highly impactful specs as a result were changes I somewhat anticipated and expected. On top of that, I could feel some unexpected positive changes as well.

#2. The Unexpected Changes

A. Increased psychological safety in team operations

Judging the impact clearly increased psychological safety, knowing we didn't have to waste engineering resources. Additionally, it freed us from the endless pressure of having to quickly tackle the piled-up backlog. Since we had already estimated the impact sizes for that backlog, the prioritization sorted by impact brought a weird sense of peace to my mind.

B. Exchanging helpful feedback more smoothly than before

When giving and receiving feedback on the impact of each product spec, we became able to do it more efficiently. Because the target user count and impact were already calculated based on data, the persuasiveness increased, and unnecessary "I think it might be like this or that" arguments decreased. Also, when discussing the assumptions applied throughout the process, these assumptions were now in highly specific areas, so there were fewer cases of massive disagreements. For example, while we used to debate the expected increase in the final conversion rate, now we discuss each element that makes up that expected increase (first entry rate, button click rate, etc.). Furthermore, feedback in a narrow scope is much more practically helpful than broad feedback. (Saying "I just don't think they'll swipe left and right much here because..." leads to a more meaningful exchange of ideas than saying "I ultimately don't think they'll pay much because...")

C. Easier to prioritize from a holistic product perspective

Each squad is sprinting to improve the target metrics they're responsible for (like "ad revenue," "subscription revenue," or "specific engagement rates"). Because of this, there's always an underlying anxiety about whether the product is being improved in the optimal priority order from a total sum perspective. However, as we started discussing impact and became able to convert and compare impact figures into upper-level metrics for each target metric we were handling, that anxiety subsided. For instance, specs aiming to increase a specific engagement are ultimately evaluated by converting how much they affect product retention, while specs aiming to increase ad/subscription revenue are evaluated by converting how much they affect product LTV. With prioritization becoming far more flexible than the previous quarter, it even enabled literally hyper-flexible team operations where sprint planning and engineering resource allocation are coordinated across multiple squads.

The good changes mentioned above are actually still in the "process" phase. Only when the product specs created this way actually generate huge impact in the end can we call it a complete transformation. We've roughly selected the impactful specs that will move into development for the rest of the quarter. How will the results turn out? I'll have to wrap this up with a final follow-up post on our impact discussions at the end of the year.