Subscriptions and Price Testing (Part 2)
🐮 Intro
In my previous post, I shared the preparation process for the price testing experiment in detail.
I hope the previous post gives you some helpful insights into experiment success criteria and experiment design.
In this post, I’d like to share the experiment results and our learnings.
The results were actually shocking?, which led to even more learnings for our team.
🧪 The Experiment Process
The experiment ran for quite a long time. We ran iOS for 6 weeks and Android for 7 weeks.
Due to team resource allocation, the iOS experiment kicked off first, followed by Android.
We targeted the top 3 countries that have a major impact on subscription revenue,
and created a total of 4 groups per platform for the experiment.
Targeting new users, the sample size for each group was about 50,000 for iOS and 15,000 for Android. Therefore, the total number of experiment participants was roughly 200,000 on iOS and 60,000 on Android.
We ran the experiment by creating a total of 4 price sets.
🌝 Group A: Monthly(Ma) + Annual(Aa) (Control)
☀️ Group B: Monthly(Mb) + Annual(Aa)
☀️ Group C: Monthly(Mc) + Annual(Ab)
☀️ Group D: Monthly(Md) + Annual(Ab)
😱 The Experiment Results
The analysis of the iOS results, which started first, was completed in the second week of July.
The result showed that Group C had a 15% improvement over the control group, so we baselined Group C. The Android results, which started later, finished analysis a bit later in late August.
Unlike iOS, the control group won on Android.
iOS Winner: Group C,
Android Winner: Group A (Control),
Sigh….
This was everyone's exact expression when the results came out
Because the results came out differently, we had no choice. We decided to go with different prices for the two platforms.
Intuitively, our team had concerns about the same service having different prices across platforms.
We had deeper discussions with our teammates regarding this and the experiment results.
The learnings we got from that process are below…
Still, since the iOS MRR portion is quite large, baselining the experiment winner showed a positive effect on our overall MRR.
🐮 Learnings
First off, while the goal of both services is identical in that they aim to “provide a successful morning,” there are a few areas where the UX of the two mobile platform apps isn't unified.
Also, there are features provided differently depending on the characteristics of each platform. (For example, prevent phone turn-off, prevent app uninstall, display over other apps for the alarm dismiss screen, etc.)
You might think, “How can one service have different UX just because the platforms are different?”
But I believe this can vary depending on the philosophy of what value you want to provide to the product and the user.
We recently saw an example of optimizing a service for platform characteristics from Google as well.
(Google decided to ditch Material Design in its iOS apps and utilize UIKit and AppKit instead…)
Additionally, among the many experiments we ran before, some product specs yielded different results across platforms and were baselined as-is, which partly explains the UX discrepancies. (For experiments, we aim to unify them by building follow-up hypotheses that can both yield better results than before and unify the UX.)
There were also some differences between platforms regarding premium features.
Of course, aside from platform-specific traits, our team has constantly been thinking about how to unify things, and as a result, we are currently embedding this through ADS (Alarmy Design System).
In conclusion, we confirmed that we were conveying the value of premium features differently across the two platforms.
iOS was pitching it much more proactively than Android. Because of this, we realized through this price test that from the perspective of users actually experiencing the product's value, the two platforms were being perceived differently.
🔥 Follow-up Actions
The direction the Subs squad chose as a result of the price test was:
Unify the differences in premium features and value delivery across both platforms
Then unify the pricing
So, the Subs squad spent about a quarter working on unifying the premium value delivery.
Currently, we have unified over 90% of the premium features and value delivery between the two platforms.
💡 Conclusion
The price test started with the goal of increasing MRR, but
interpreted from another angle, I felt it was like taking a test where we are evaluated by users on the paid features we currently provide.
Because the experiment result is precisely the value that users have acknowledged in the market.
Through this experiment, the Subs squad was able to focus more on where our delivery fell short, and as a result, we gained the seeds for a new growth loop.