Are the Numbers You're Looking At Actually Okay?

When defining a data analyst's role, and asking analysts what they expect of themselves, the first answer that probably comes up is "finding insights from data and reflecting them in decisions." Even though analysts actually wear many hats, the reason we emphasize this is because it's a domain requiring context and critical thinking, meaning it's not easily replaced—and it's where analysts feel their reason for being most strongly. Experiencing your analysis actually influence a decision and move the metrics is a kind of fulfillment hard to find in repetitive tasks.

What is an "Insight Used for Decision Making"?

But here, we need to make a distinction. "Finding insights" and "finding insights used for decision making" are not the same thing.

The caveat "used for decision making" carries several implications. Is it actionable? Can we expect a sufficient impact? Is the cost affordable? Is this information needed right now? Without these conditions, an insight is prone to end up as just an interesting piece of data, met with reactions like "So what?", "The impact is too small," or "We don't need this right now."

Judgment on Facts Over Facts Themselves

Taking it a step further, what really matters for an analyst isn't the fact itself, but the judgment on that fact.

"The conversion rate is 95%" is a fact. But this fact alone makes nothing happen. Whether this conversion rate is high enough or too low—that judgment must be attached before decision-making can even begin. Depending on whether that 95% is high compared to the industry average, dropped from the previous quarter, or should be higher given the product's nature, the exact same number could mean "we are doing great" or "we have a problem."

Looking at data and organizing facts is just the beginning; judging what those facts mean is the essential role of an analyst.

The Difficulty of the Act of "Finding"

The process of finding insights starts with a hypothesis and verifying data one by one. It sometimes feels like sifting for pearls in the sand. There are two reasons why this isn't easy.

Establishing a hypothesis itself is vague. You can't just dig anywhere; you have to decide where to dig first, which requires prior data and context. In reality, we often start off in a certain degree of vagueness.

There is a dilemma where the deeper you dig, the smaller the impact gets. Surface-level data is already known to everyone, so you have to go deeper to find new facts. But as you go deeper, the target audience narrows and the impact shrinks.

Considering these difficulties, we need to take a different approach to insights. If we divide insights into two types:

• Discovery — Finding and presenting an unknown fact

• Judgment — Pointing out an existing fact as a problem

I place more weight on judgment between the two. The sense and perspective to say, "This is a problem," amidst data accessible to anyone. I think this is one of an analyst's most important qualities.

Many organizations ask new hires for their opinions on the service or interview them. This is because things existing members are so used to and take for granted can be discovered through the unbiased, pure eyes of newcomers.

"Why is it set up like this? Has it always been this way? Is this the right way, then?"

It's natural for this sense to dull over time. That's why intentionally asking these questions is key. We need to question once more whether the facts we knew and our understanding of them are actually correct. Since judgment is context-dependent, what wasn't recognized as a problem in the past might not have seemed like one back then. When the situation changes, the same facts read differently.

1. Finding a Baseline for Comparison

For judgment to work properly, there must be a baseline for comparison. Judgment without a baseline is closer to a gut feeling, and it's hard to persuade with just a gut feeling. (I find it hard to even feel this sense myself.)

Let's take an example. We added a screen requesting a new system permission from the user. If the permission grant rate on this screen is 90%, should we consider it high enough and problem-free? It's hard to judge with this number alone. But if we have a baseline, the story changes. If there's an already existing permission request page in a similar location, we can refer to that page as a baseline. If the baseline is 97%, we can be fairly confident that there is room for improvement in the new permission's grant rate.

Here is another example. There were screens during the initial onboarding flow where the conversion rate dropped significantly. The commonality among these screens was that they required network communication with the server. The hypothesis that users were dropping off due to network delays or errors seemed quite convincing to explain the phenomenon.

• Page 1 conversion rate: 60% (**Arbitrary example value)

• Page 2 conversion rate: 60% (**Arbitrary example value)

At first glance, the problem seems clear and actionable. But thinking carefully, something is off. If it's a sequential experience from Page 1 -> Page 2, the users who experienced network issues on Page 1 have already been filtered out, meaning the users who reached Page 2 are those with a higher likelihood of conversion. Nevertheless, the fact that Page 2's conversion rate is similar to Page 1's implies there might be other drop-off factors on Page 2 besides the network. You need to have that sense of "something is off." In fact, by comparing the conversion rates of the two pages, we were able to discover quite an important UX improvement point.

When you have a baseline to compare against like the above, it becomes much easier to judge whether there is a problem or not.

2. Increasing Understanding of the Service

You can't judge by looking at numbers alone. You need to understand the context, business, users, and platform those numbers sit on to make a meaningful judgment.

For example, if the alarm dismissal rate differs between Android and iOS, should we judge that there's a problem with a specific platform? The way alarms ring on Android and iOS differs according to each platform's native policies. The policies are as follows:

• Android — When an alarm rings, the alarm dismissal screen is automatically shown over the lock screen.

• iOS — You have to unlock the lock screen and manually enter the app for the dismissal screen to show.

If you compare the initial ringing drop-off rate of the two platforms without knowing this difference, it's easy to conclude that one side has a problem. But in reality, it could just be a natural result stemming from the difference in platform policies. The physical difficulty of reaching the dismissal screen is higher on iOS, and a lower dismissal rate caused by this seems quite convincing. If you know this context, you can distinguish whether the drop-off rate difference is within an explainable range due to platform characteristics, or if there's a problem beyond that.

Understanding the domain prevents you from misreading data like this, while simultaneously serving as the starting point for more accurate judgments.

Looking with a Suspicious Squint

I feel like I've been talking a bit vaguely. But I wrote down parts I feel are quite important for an analyst's competencies these days. I described finding a comparable baseline and increasing the understanding of the service somewhat obviously, but honestly, I'm also not sure how to develop this sense—the ability to recognize a problem as a problem. It might be because it seems closer to a universal, general competency rather than a specialized skill. Defining a good analyst, a capable analyst, and developing those abilities still has a lot of ambiguous corners for me. If anyone knows a good way, please share.

The competencies expected of an analyst will diversify and change with the times, but my belief that the essence will always remain similar is unchanged. I dare guess many analysts share similar thoughts. I've come to think that "finding insights and reflecting them in decisions" relies more on the ability to recognize things rather than the pure skill of the act of finding something. Based on this thought, efforts to periodically check whether the currently given data and the numbers right in front of our eyes are actually plausible seem to refresh a rather confined perspective. At first glance, it's easy to just pass by. But if you really sit down and look into it, won't you see problems you hadn't discovered before? "Making an issue out of it makes it an issue" is an expression usually used in negative situations, but sometimes, if you look at the data with a suspicious squint, treating the current state as a problem, I think you might discover unknown facts at unexpected points and improve the service.