[Alarmy Android] A Dynamic Subscription Launch

*This post is a Medium adaptation of the notes I presented on Notion during our town hall meeting.

Throughout the first half of 2020, we sprinted to launch the Alarmy Android subscription model. Adding features and implementing the server for subscriptions was our Q1 OKR, while prepping for the global subscription launch was our Q2 OKR.

To cut to the chase, as of late June 2020, Alarmy Android has been offering its Premium (subscription) model to users worldwide.

In January, my main task was launching Alarmy's subscription model, Premium. According to our January goals, the plan was to launch Alarmy Premium for US and Korean users in March, then roll out additional features and a global launch around April or May.

Implementing the server for Android subscription authentication using GoLang and integrating it with the client went off without a hitch. And the Premium features—Wake Up Check, Backup Sound, Time Pressure, and Typing Mission—were developed one after another. Aside from some schedule delays, it seemed like we could smoothly unveil the Premium features to our users in late April with zero issues.

The Beginning of the Incident

On April 21st, we realized there was a problem when a Slack inquiry came in from the Product Operation group.

I checked the Google Play Console to see what was going on.

In the Google Play Console, refunds had been steadily increasing since April 5th, and the numbers looked abnormal, so I contacted the Google Play Developer Support team.
I hoped reaching out would give me a clue, but the support team's response was incredibly vague: "It occurred due to buyer error or seller issue." (So it's everyone's fault???) Along with a note saying it seemed to be a user-side issue, I was left with an ambiguous conclusion, wondering if I just had to wait it out.

Above all, figuring out the root cause was even harder because I was under the illusion that no direct changes had been made to the payment-related code.

But...
I got a sinking feeling when a message in the Google Play Console—"We detected instances where your package didn't acknowledge in-app purchases or subscriptions. Starting with Billing Library 2.0, all purchases will be automatically refunded if not acknowledged within 3 days."—overlapped with a phrase I'd seen in the developer docs during the subscription implementation process. I immediately combed through the Git commit log history.

I had merged the implemented subscription module for partial testing, and as a result, the Billing Library was bumped up to the latest version used for that subscription module, which started affecting the existing code. (A harsh lesson in the importance of distinguishing between api and implementation in Gradle dependency options.)

Consequently, the Billing Library updated to the latest version. Even though I hadn't modified the code, the library's behavior changed, requiring admin acknowledgment for payments within 3 days. But because I wasn't aware of this after the code was deployed, automatic refunds had been triggering left and right.

It would have been great if I could just identify the bug, fix it, and deploy!—solving the problem instantly. Unfortunately, the timing of my discovery made things incredibly tangled, creating a whole new issue.

In the version that started rolling out in mid-April, the old Pro IAP (In-App Purchase) model was gone, and the app had already transitioned to the Premium subscription model, which was already live.
As a result, it was impossible to add and deploy logic in the app to acknowledge the now-defunct Pro IAP transactions.

Finding an Alternative

To solve the refund issue, boldly telling users, "Update to 4.21.0 and use the Premium subscription," would be easy and convenient from an engineering standpoint. However, Premium subscriptions weren't open to all users yet, and it would look like we had blocked their purchases just to push subscriptions.

While looking for a workaround, I discovered that I could acknowledge purchases using the API I'd used when building the subscription authentication server—as long as I had the purchase token visible in Google Play Console > 주문관리.

I developed a script on my local server to handle the acknowledgment process, tested it on a token that was about to expire, and verified that no refund occurred even after the deadline passed.
(I survived😅)

A 5-Week Journey

Once I found a way to patch things up, I didn't look for a deeper solution, leaving only a path of manual drudgery ahead.

From the moment I identified the cause and set up the local server until the 100% subscription rollout in late May, I committed to the grind: every day at 11 AM, I'd copy Pro purchase tokens from the console's order management and manually repeat the authentication process.

I created an #android_order_management channel in Slack to receive regular notifications and manage the daily tokens and their processing status.
During that period, I manually processed 100+ Pro transactions.

While juggling these Pro payments, we launched Premium for US and Korean users in late May, and successfully completed the global Premium launch by late June.

Due to policy changes, anyone looking for the old Pro IAP model is now directed to the Premium subscription.

Conclusion

There is no such thing as an unfounded VoC (Voice of Customer), and version control must be handled with care.

You can also find more posts in our ‘Brunch Magazine’.