Subscription Price Testing (1)

🐮Intro

We're already in the final sprint of Q4 2021.

Leading the Subs (Subscription) team, I ran a ton of experiments again this year.

Most of these experiments were aimed at "increasing free trial conversions" and "boosting the conversion rate from free trial to actual payment."

Some prime examples include an experiment (post) on raising the mental accounting threshold through onboarding, improving the purchase page to better highlight premium value, and sending messages right after installation via a CRM tool to deliver various user values.

One of the more massive experiments we tackled was subscription price testing. Since there's relatively little external information available on price testing, we went through a lot of agonizing and trial-and-error during the process.

When preparing for a price test, there are many factors to consider. I hope our case (and our blunders) can serve as a helpful reference so you can run easier, more refined price tests. (It would be great to share other cases with each other too. 🐮)

💰An Experiment You Can't Run Just Any Day: Price Testing

In particular, compared to other experiments, price testing wasn't something we could just pull off on any given day.

Since launching our subscription product, we had made numerous improvements to premium features and consistently added new ones, so the team decided to move forward with the test, confident that we were providing much higher user value.

A crucial metric in the subscription business is MRR (Monthly Recurring Revenue) (understanding basic subscription monetization metrics is nicely summarized here).

The sole objective of this "price test" was to increase the MRR metric.

There are three main levers to increase MRR:

• ☀️ Increasing Acquisition

• ☀️ Increasing Retention

• ☀️ Improving Monetization

Among these, as I learned from a previously written post, Pricing 101,

the order of highest business impact when improving each lever is "Monetization > Retention > Acquisition."

Therefore, we decided to proceed with "price testing," a method for improving monetization, which carries the highest business impact.

🧪 Experiment Design 1: Hypothesis and Success Criteria

First, an experiment needs a hypothesis and success criteria.

Our hypothesis was: "Providing a more attractive price set to new users will increase MRR."

The success criterion was "the group with the highest expected total payment amount per 1,000 new users." Formalizing this success criterion looks like the formula below.

공식 = 1000 * Trial CVR * T2P CVR * (1 — Churn Rate) * Price

Breaking this formula down a bit,

we wanted to look at the total revenue from new users who install the app, convert from a free trial to a paid subscription, and continue using it without requesting a refund.

Since Alarmy offers both annual and monthly subscription products, the sum of the amounts calculated by applying this formula to each product had to be the highest.

합산 매출 = 월간 * 공식 + 연간 * 공식

Ultimately, the experimental group with the highest "total revenue" would win.

As we were designing the experiment, our biggest concern was naturally that,

as you can see from the formula, some metrics were lagging indicators that could only be collected after a certain period of time. And since we were only testing on new users...

the experiment duration dragged on, and we needed a massive sample size.

(If we determined that the Churn Rate in the formula was negligible, we could have excluded it, which would have let us run the experiment much faster. However, at the time, our team took a conservative approach, knowing that once a price is set, it significantly impacts future MRR. As a result, we ended up making our decision based on every metric represented in the formula.)

In reality, we ran the experiment for 6 weeks on iOS and 7 weeks on Android,

involving only our top 3 major countries with a total of 4 experimental groups.

We allocated about 50k users per group on iOS, and about 15k users per group on Android.

🧪 Experiment Design 2: Setting the Experimental Prices

One of the points we agonized over the most in the price test was how to structure our price tiers. We utilized learnings from our previous Pricing 101 and researched how other external apps structured their prices.

Alarmy's subscription products include annual and monthly plans.

When users enter the purchase page, they process three things regarding the price:

• Annual price

• Monthly price

• The relationship (discount rate) between the annual and monthly prices

Taking these three factors into account, we set the price variables for each product as follows:

• Monthly: Ma, Mb, Mc, Md

• Annual: Aa, Ab

Using these defined variables, we created a total of 4 price sets.

• Group A: Monthly (Ma) + Annual (Aa)

• Group B: Monthly (Mb) + Annual (Aa)

• Group C: Monthly (Mc) + Annual (Ab)

• Group D: Monthly (Md) + Annual (Ab)

While creating these price sets, our two main considerations were:

• ☀️ Range of the monthly price (min, max)

• ☀️ Discount rate of the annual plan compared to the monthly plan

Specifically, when we researched apps with similar characteristics, the discount rate for annual versus monthly plans was usually around 30% to 50%. We kept that ratio in mind when building the price sets for our test.

Once these sets were finalized, our iOS and Android engineers would create the necessary subscription SKUs and prep for the price test.

💡NOTE: Looking into the concept of Subscription Groups

When creating SKUs, there are some differences between how iOS and Android manage them, which is good to know.

On iOS, there's a concept called a Subscription Group that bundles and manages SKUs when they are created, whereas Android does not have a separate Subscription Group concept. So for Android, the service provider has to build the grouping logic between SKUs themselves.

Originally, the Subscription Group on iOS was introduced to solve two UX issues.

First is automatically handling upgrades and downgrades between subscription products for the user. Previously, users had to cancel their current product, wait for the cycle to end, and then purchase again.

Second is providing a system to prevent users from accidentally subscribing to two products at once. Without groups, unintentional double subscriptions used to happen.

As helpful as the Subscription group is, it can sometimes get in the way during price tests. If you create SKUs in the same subscription group, a downside is that users can later see the other price tiers in their subscription settings.

p.s. I'll cover subscription groups in more detail later on.

Experiment Results 🔥

I was about to write up the experiment results, but seeing as this post has already gotten too long,

I'll share them in the next post.

At DelightRoom, we are constantly pursuing the essence of our product and building it by working through various challenges together. We are looking for someone to join us in the zero-to-one experience of redefining Alarmy from a simple time-telling tool into a service that takes responsibility for your routine from before bed to after waking up! Please check out our hiring page.