Product Specs
To flesh out an idea and start working on it, we usually write a product spec (at DelightRoom, we call this a PBI, or Product Backlog Item). This spec includes a variety of details, but today I want to talk specifically about "Impact."
One of the key factors in deciding the priority of a task is generally the size of its impact. Among the many problems to solve within a service, tackling them in order of impact size is undeniably an effective way for a team with limited resources to operate.
For instance, improving a feature that affects a vast majority of users in the service can be considered a high-impact task. Between "improving the UI/UX of a page seen by 1,000 users a day" <> "improving the UI/UX of a page seen by 10,000 users a day," choosing the latter has a greater impact because it improves the experience for a much larger group of users.
However, there are situations where simply reaching many users doesn't always guarantee a large impact. A minor issue that many users experience but doesn't severely hinder app usage has a very different level of gravity compared to a critical issue that drives users to abandon the app entirely.
A page seen by 1,000 users a day, causing 50% of them to bounce (500 users lost)
A page seen by 10,000 users a day, causing 1% of them to bounce (100 users lost)
In the example above, if we have to take immediate action, prioritizing the former would be more effective from an impact perspective.
We want to rationally predict the effort invested in a task and the resulting outcome. When we look back at cases where we put in a lot of hard work but the metrics didn't improve as expected, it was often because we either didn't predict the impact beforehand or neglected to do so. In most cases, we would have realized that the task's efficiency was low—meaning the output wasn't great compared to the input—if we had just calculated it in advance, but we skipped that process. Through these experiences, our members naturally internalized the importance of estimating impact, and it has become a crucial part of our product specs.
Can we compare them? Yes we can!
While everyone agrees on the importance of "impact estimation" and shares a strong enthusiasm for it, it can sometimes feel like we're out of touch on exactly how to compare things. Service development usually consists of various specs and tasks with different natures, and there are ambiguous moments when figuring out prioritization because these tasks are so fundamentally different.
It's hard to view 1) fixing an existing bug in the service and 2) improving the UX/UI of a specific screen as the same kind of work. Nevertheless, we still need a priority order to determine which one to tackle first. Generally, we can compare the impact of these tasks through "retention." If we could make the following estimates, we'd be able to compare them, right?
Fixing bug A will improve new user retention by 5%
Improving the UX/UI of screen B will improve new user retention by 3%
The problem is that making an accurate prediction before doing the work is difficult. Just how much tasks A and B will help improve retention is impossible to know without actually executing them. It would be great to run an experiment, but even experiments require engineering work, which eats up resources. For us, trying to prioritize tasks in the planning stage before committing any resources, the option of running an experiment is meaningless.
Usually, we calculate estimates by appropriately combining existing data and intuition (using a fancy term, this is called a "Fermi estimate").
The number of users experiencing bug A is N, and the retention difference between users who encountered the bug and those who didn't is K%. Therefore, fixing bug A will improve retention by P%.
The number of users viewing screen B is K, and the bounce rate on this screen is D%. When the improved UI/UX is applied, based on similar past cases, there will be an W% improvement in the bounce rate -> The UI/UX improvement of screen B will yield an E% improvement in retention.
I explained this using a simple example, but in practice, predictions are made using a similar format and structure. On top of translating impact into comparable numbers, it's also important to attempt to improve prediction accuracy by creating a feedback loop that compares the initial prediction with the actual changes afterward (as I kept doing this, I felt my predictions were getting closer and closer to reality).
Comparing Impact Expressed in DAU
Let's define the current service's stats as follows.
User acquisition => Average 5,000 users per day
Retention Curve => (D0 : 100%, D1 : 50%. D2: 40%, .. D6: 16%)
We can estimate the DAU for N days from now.
Ultimately, DAU is calculated as the daily average newly acquired users x the AUC of the retention curve. Based on the example above:
DAU after 6 days : 5,000 * (D0 + D1 + D2 + D3 + D4 + D5 + D6) = 11,700 users
DAU after 30 days : 5,000 (D0+D1+…+ D30)
DAU after 90 days : 5,000 (D0+D1+……. + D90)
DAU after 365 days : 5,000 (D0+D1+…………+ D365)
Now let's assume a scenario where we can make a comparison.
If the current daily average user acquisition changes from 5,000 → 10,000
If the current retention curve increases by 10%
1) The increment when the daily average user acquisition changes from 5,000 → 10,000
Size of DAU increment : 10,000(D0+D1+ .. +Dn)-5,000(D0+D1+..+Dn)
2) The increment when the current retention curve increases by 10%
Size of DAU increment : 5,000(D0`+D1`+ .. +Dn`)-5,000(D0+D1+..+Dn)
By comparing the sizes of these increments, we can estimate which lever will have a bigger impact on DAU growth. We can also predict which strategic decision will be more helpful for the service's growth.
While the organization carries out tasks of various natures, if you look at their ultimate goals, they all boil down to two top-level objectives: "traffic (DAU)" and "revenue." At first glance, seemingly unrelated tasks still contribute to improving these top-level metrics. As a result, even two superficially different tasks can be reduced to and compared against these overarching metrics.
It's also interesting to see a case where retention was converted into our top-level metric, revenue, for comparison. I recommend checking out the post below. (Retention is King ..? by Stephan)
Wrapping Up
Predicting the impact of a task is one of the crucial elements of the development process. Of course, refining the prediction techniques is necessary, but the most important thing is the very act of "making a prediction." We need to look back and see if we haven't neglected the process of quantifying and examining a task's effect under the excuse of being too busy or due to overconfidence in our intuition.
As a data analyst, despite being in an environment where I can access data easier than anyone else, there were times when I neglected the prediction process for various reasons. As a result, situations occurred where the resources of highly skilled personnel were wasted.
Ultimately, making predictions must become a habit and part of the process. We must not forget that it is a task just as important as running analyses and drawing insights.