Apple App Store Product Page Optimization (1)

Let's finally do A/B testing on the App Store. Enough with the before-and-after tests...

The App Store finally got an experiment module.

It's a feature similar to Google Play Store's "Store Listing Experiments,"
now available under the name "Product Page Optimization (PPO)."

Usually, my blog series style is to start by explaining why this highly anticipated experiment module is so precious and important before getting into the details. But right now, the sheer refreshing relief of finally having it is so overwhelming that, just this once, I'll dive straight into the main topic and save the background context for a follow-up post.

The Arrival of PPO

For those of us in charge of organic user growth, a store experiment module is desperately needed. Before-and-after tests have massive limitations when trying to refine user acquisition strategies based on meaningful hypotheses. Because of this, we end up pouring far more time into the Google Play Store, which offers an experiment module, than the App Store, which didn't. Sure, Google's experiment module has its flaws—like not showing detailed metrics—but simply declaring a winner based on the ultimate metric of organic installs is still something to be incredibly thankful for.

Then, amidst all this, Apple announced that they planned to launch a store experiment module (PPO) in the App Store by the end of 2021.

I quietly rang in the New Year and logged into the store console, but for the life of me, I couldn't find the entry point for the store experiment module. According to the guide, it was supposed to appear on the bottom left... There weren't many articles about it either, and most of them used phrases like "It seems..." or "It might...," giving off a strong vibe that they were just guessing too, which wasn't very reliable.

But it's always darkest under the lamp. I eventually found a window to apply for something right on the official guidance page.
To be precise, the name of this experiment module is Product Page Optimization (PPO), and this application portal was for signing up to get early access to the massively overhauled app submission process, which included PPO.

Desperate to use it even a month earlier, I applied right away, and it was approved shortly after.

And just like that, the entry point appeared. (Heart pounding)

Differences and Similarities with Google Play Store Listing Experiments

To begin with, the two stores have entirely different configurations for listing assets. For instance, the App Store has a Subtitle (30 characters), while the Play Store has a Short Description (80 characters). Furthermore, when and how each asset is displayed differs, and this is constantly changing within each platform as well. Assuming we're all familiar with those basics, I'll briefly summarize the differences below. (Since this is only based on what is currently known, there might be some missing details.)

a. Store Assets

1) App Name cannot be tested on either platform

• That's right. The App Name is impossible to test on both OSes.
  But since the App Name is a variable where changes can be observed fairly clearly even with before-and-after tests... it's understandable!

2) Text cannot be tested

• Unlike the Google Play Store, where you can run experiments on the Short Description and Long Description, the App Store does not allow experiments on the Subtitle or Description.

• In other words, you can only run experiments on the icon, screenshots, and app previews (videos).

3) For icon experiments, the variant image file must be included in the app

• Unlike the Google Play Store, to run an icon experiment on the App Store, you must place the variant image inside the app binary in advance.

• To put it differently, to run an icon experiment, you have to include the variant icon in the app binary, pass the app review, and only then can you set up the experiment.

b. Relation to App Review

1) It goes through the same review process as an app review, but the app itself isn't being reviewed

• In the Google Play Store, as long as you meet the specs (character limits, file sizes, etc.), an experiment is applied immediately upon setup. The App Store requires a separate review process, and the experiment only starts after it's approved.

• It's effectively identical to the app submission process. The only difference is that the target of the review is the store listing, not the app itself.

• It takes way, waaaay less time than an app submission review. (It took 20 minutes.) This is probably because there aren't a ton of active review requests coming in yet.

• Therefore, you don't need to worry—even if this review gets rejected, it has zero impact on your live app.
  (The message in the pop-up window above is slightly misleading and can definitely give you a little scare.)

2) Cannot be done simultaneously with an app review

• If you have submitted an app and it is currently under review, you cannot request a PPO review.

• Conversely, if you request a PPO review, get it passed, and have an experiment running, submitting a new app for review in the middle of it will halt the PPO experiment. In this case, you cannot resume it; you have to gather sample data all over again from scratch. Therefore, you must plan your experiments while keeping your app release cycle in mind.

c. Number of Experiments

1) You can run multiple countries simultaneously in a single experiment, but you cannot analyze the data separately

• When running localization experiments on the Google Play Store, you can only set up one country (language) per experiment. On the App Store, you can set it up simultaneously for all countries (languages) where the app is distributed. In doing so, the screenshots don't just apply as a single blanket variant across all countries; you can set up individual variants for each country. For example, if you're setting up a variant that adds "Screenshot A," you can set up translated versions of "Screenshot A" as variants for each specific country.

• I thought this was a massive win, but when I went in the next day to analyze the metrics, I realized it doesn't break down the data by country...

• Beware of this trap. You can run multiple countries at the same time, but you really shouldn't...

2) Only one experiment at a time

• The Google Play Store lets you run up to 5 localization experiments simultaneously. The App Store lets you run exactly one.

• Even with the Google Play Store's 5 experiments, they had to be configured for 5 different countries (languages). For example, you couldn't run a screenshot experiment and an icon experiment for the US (English) as two separate experiments. I was always secretly bummed about that, but the App Store just straight up won't let you set up 2 or more experiments at all.

Other Limitations Worth Knowing

• PPO experiments only apply to users on iOS 15 or higher. Depending on your user base's OS version distribution, it might not be very useful.

• To run icon experiments, your app needs to be built with Xcode 13 or higher. Fortunately for Alarmy, this was already the case.

Honestly, the hardest part right now is that because it's not being actively used in Korea yet, there's a lack of (or basically zero) reference cases to look at. The clunky translation within the interface (e.g., naming a "variant" as "treatment") and the presence of misleading touchpoints throughout the entire journey are also among the challenging aspects. Despite all this, simply having an experiment module where we can stack up clear lessons is incredibly exciting and fun. From an experimentation standpoint, time is literally an asset, so I'm currently prioritizing running experiments on hypotheses that are worth executing quickly, even if they're a bit rough around the edges. As for the "analysis" feature where we extract insights, I plan to write about how it differs from the Play Store in follow-up posts based on specific examples.

Meanwhile, the app preview experiment I set up yesterday didn't go as planned.

Yesterday was full of short but impactful lessons!

So, the next part will be,

Apple App Store Product Page Optimization (2) — Setting up App Preview Experiments

mail : stephan@delightroom.com
instagram : @seosinghan