How Often Does Alarmy Get a Health Check? (feat. Ad Metrics)

Efficient ways to monitor metrics

These days, it seems like a lot of companies offer annual health check-ups as part of their benefits package. Personally, it's one of my favorite perks. Getting objective numbers on how my body changes as I age, right at that sweet spot of a one-year interval, serves as a wake-up call and really helps me figure out how to take better care of myself moving forward. (Though it is pretty sad seeing my body fat percentage and muscle mass get worse every year ㅠ)

Products are much the same. A product health check is absolutely essential because it lets us understand its condition through objective numbers rather than just gut feeling, helping us figure out how to improve it. When we talk about a product health check, we mean looking at changes in KPIs and related metrics through data. Tracking specific metrics to measure whether the product is shaping up the way we intended is widely recognized as a mandatory process for building a healthy product.

At DelightRoom, we usually hold recurring "Metric Meetings" within each squad to regularly check on the squad's health.

There is no silver bullet, but...

There is no such thing as a one-size-fits-all meeting. You just need to look at the data appropriately based on your product's specific situation and conditions. However, if I were to offer a guideline, wouldn't it be best to align your metric reviews with the product development cycle? If a product is developed on a monthly cycle, spending an hour every single day analyzing metrics might not yield much value. Why? Because if no changes happened on the product side on a daily basis, there's a high probability that no major shifts occurred in the product's performance either. Even if you look at the data, it likely won't look much different from yesterday. Conversely, checking the metrics just once a quarter feels far too loose. You'd likely miss out on tracking various changes that happened over the course of three development cycles (1 month x 3) right when they occur.

(System metrics, like server performance, are an exception, of course. Those metrics are tied to the normal operation of the service and absolutely must be tracked in real-time!)

Ultimately, it comes down to resources and marginal utility

I don't think anyone would disagree that the absolute best approach is looking at the data as often as possible. The problem is resources. What we end up wrestling with is ensuring that the marginal utility of the resources we invest at any given moment meets our expectations. Since the utility of investing resources tends to experience diminishing returns, our goal should be finding that sweet spot of just the right amount of resource allocation.

If we broadly categorize how we allocate resources for data exploration, we can break it down into two aspects: 1) Frequency of exploration, and 2) Intensity of exploration. From a frequency perspective, checking data more often has the advantage of letting you catch shifts in metrics agilely. On the flip side, the amount of new information you gain compared to what you already knew decreases, which means the utility relative to the resources spent drops as well.

Regarding the intensity of exploration, diving deep is essential for identifying the root causes behind surface-level metric changes. If the intensity is too weak, you'll likely stop at merely checking superficial or lagging metrics. Uncovering the exact reasons behind a change absolutely requires investing a certain amount of resources per monitoring session.

There's also a right way to monitor based on your specific objective. For example, if your goal is to spot issues rapidly, it's crucial to crank up the frequency of your checks so you can catch major shifts fast. In contrast, if your goal is to investigate and brainstorm the root causes of an issue, you need to set aside enough resources when looking at the data, think deeply about the phenomena, and build a logical explanation.

The nature of ad metrics

One of the defining characteristics of the ad market is probably how fast the ad landscape changes. Not only is the market's feedback on our actions incredibly swift, but market shifts completely unrelated to a publisher's actions are wildly unpredictable day to day. So, as someone checking the data daily, it's true that every day brings a mix of excitement and anxiety. While product development moves unilaterally from product → user, the ad monetization ecosystem is heavily impacted by the real-time actions of various stakeholders being reflected in the product, making it quite hard to predict..

Publishers, advertisers, ad mediation platforms (Mediations, SSPs, DSPs, etc.), and user feedback all converge to explain the fluctuations in ad metrics.

Because of this, from a publisher's perspective, we frequently see ad metric changes where the exact root cause is totally opaque. Sometimes it shifts because of an advertiser's intent to display certain ads, or maybe the attractiveness of our ad inventory changes during the programmatic bidding process due to some microscopic tweak. Unknown variables pop up very, very often on the publisher's end.

DelightRoom's monitoring process

Since ad metric trends fluctuate by the day, quickly detecting changes is an absolute must. While the root causes are often not intuitively obvious, the impact of those changes is highly visible (e.g., changes in revenue). This means we can catch the massive shifts even if a single monitoring session doesn't consume a ton of resources.

Once changes are detected, we need to share them internally and start digging into why they happened. However, not every single change requires an immediate deep dive. Depending on the importance of the shift, we might react agilely or choose to observe it with a bit of a buffer.

In the ad monetization squad I belong to, we look at metric changes across three different layers.

1. Daily Anomaly Report

When breaking down the minimum unit for monitoring, we usually divide it based on homogeneity/heterogeneity. For instance, we might observe changes grouped by ad format (banner, native, rewarded, etc.), or even go a bit more granular and break out our monitoring units on a per-ad placement basis.

Currently, our squad sets the minimum monitoring unit at the intersection of platform x ad placement x major country group (~mediation group). For example, "iOS US Placement A", "Android US Placement B", and "Android Korea Placement A" are treated as entirely separate monitoring units.

The problem is that to "properly" grasp shifting conditions, we have way too many segments to keep track of. If we lower the resolution—meaning we reduce the number of monitoring units (e.g., just checking metrics by platform)—crucial pieces of shifting data get diluted. Conversely, slicing it up too finely leads us right into the issue of spending way too much time on it. I believe this is exactly where automated monitoring comes into play.

Through anomaly detection, we systematically check for daily shifts in traffic and major revenue changes. Ultimately, the most important thing is the change in revenue. The impact of a shift is often directly equivalent to the sheer size of the revenue change, so just by looking at the magnitude of revenue shifts, we can catch the most critical items to some extent.

2. Daily Manual Monitoring

Every day, our members dedicate a set amount of time to review metric changes together. You can think of it as a process where revenue changes are flagged for our tracked monitoring units via the Anomaly Report, and for the highlighted items, our members dive deeper into the data and discuss to uncover the root causes behind those revenue swings. Along with the goal of sharing these shifts, we debate their importance to decide if it's something worth spending extra resources on to investigate further.

Because the anomaly detection serves as a first pass, filtering down the monitoring units that actually need our attention, we get the massive advantage of being able to manually review just a curated list and focus purely on hunting down the root causes together.

3. Weekly Review Meeting

This is a meeting where we rigorously inspect metrics and review performance on a weekly basis. While the previous processes aim to quickly check major shifts and their causes daily rather than getting bogged down in minute details, the weekly review meeting is all about wrapping up the week, summarizing the metrics, and taking a fine-tooth comb to any issues. Shifts that haven't manifested into revenue changes can sometimes slip through the cracks in the daily checks, but since we discuss the app as a whole in detail every week, we try to catch as many of the week's changes as possible. Naturally, this demands more time than our daily monitoring does.

Wrapping up

The true essence of a "health check" is quickly grasping product changes and sparking thoughtful consideration about our future direction. It would be absolutely ideal if we could go beyond merely identifying changes and root causes, and actually derive concrete action items for the road ahead. One critical thing in that process is a culture and environment where we look at data and share opinions together. Handing it all off to a single person easily leads to bias and errors. To make accurate judgments about our situation, I believe it's essential for each relevant member to look at the data and have discussions as a calibration process. At the same time, these conversations are incredibly meaningful for building psychological solidarity and motivating the people actually building the product. So, I highly recommend finding ways to increase your hands-on time with product metrics and carving out dedicated time to talk with your teammates.