App Store Product Page Optimization (2)

A/B Testing on the App Store — Setting Up the Experiment

Over the past month, excluding test-runs, I went through three experiment cycles. I thought one or two cycles would be enough to get past the trial and error, but honestly, I still feel like I'm stumbling through it. It doesn't quite feel like I'm fully utilizing it yet. Even after googling and scouring the App Store Support pages, I still can't find the answers to the question marks floating in my head. Instead, running through multiple experiment cycles and figuring things out on my own has proven to be a much faster way to find answers.

The goal of this post is to share the inevitable question marks you'll encounter while setting up App Store experiments, along with their answers. I hope that many of you, like me, will actively share your trial and error so we can use a sort of collective intelligence to supplement the App Store's lacking guidelines.

Q1.
It's stuck on "Still uploading and processing."

A1. There might be an issue with your store asset specs. If you exit and re-enter, it will tell you what the problem is. Don't worry—your existing experiment setups are saved even if you leave and come back.

This issue briefly popped up at the end of my last post. I saw it during both the app preview video experiment and the screenshot experiment. Even if you know the store asset specs inside and out, it's an incredibly easy mistake to make. For example, when exporting multiple images from Figma, you might accidentally export one as a PNG instead of a JPG. The problem is that the App Store doesn't clearly say, "There's a problem with your store assets." Instead, it just throws a "Still uploading" error, leaving the user totally confused. You want to invoke the golden rule of IT (turn it off and on again), but you hesitate out of fear that your setups will disappear, so you wait a bit longer just in case. But even after waiting half a day, the "uploading" error persists, and ironically, it's only after you turn it off and on again that it shows the correct error message.

For starters, just keep in mind that video assets shouldn't exceed 30 FPS (Frames per Second), and image assets must be JPGs, not PNGs.

Q2.
Looking at the review submission screen, my experiment setups seem to be missing. Will they not be submitted for review?

A2. They only appear missing on the display screen; in reality, they are included and submitted for review.

I completely fail to understand why this bug exists, but that's the reality. Talk about out of the frying pan and into the fire—I figured out the asset issue in a somewhat bizarre way (by leaving and coming back) and fixed it, only for something to appear missing on the very next screen. Since it looked like it was missing from the submission details, I submitted it with an anxious heart, but I eventually realized the review was processing properly. (I know this because it got rejected...)

Q3.
I'm trying to view the analysis results, but I keep getting errors.
I'm worried it's not running properly.

A3. There might actually be an error, but in my case, the analysis results magically appeared after I eventually stopped the experiment.

Usually, the analysis results weren't available for the first 1–2 days after setting up the experiment. It would be really nice if it showed a friendly error message like "Still collecting data," but instead, it either throws the error screen above or simply omits the experiment you want to see from the list, making the person in charge totally anxious. You're in the Product Page optimization tab, but only old experiments are listed, and the newly set up experiment is nowhere to be found. I submitted an inquiry as the screen suggested ("If the problem persists, click here"), but all I got was a generic macro response.

About 7 days later, I figured I had nothing to lose and stopped the experiment. Only then did the experiment results finally appear. Unexplainable bugs still lurk all over the App Store's experiment module...

Of course, the App Store is probably working hard to fix these various bugs according to their priorities, so the contents of this post will inevitably become useless someday. Error messages will get more refined, and guidelines will be updated. But from the perspective of someone running an experiment where time is money, we can't just sit around and wait for that day—we just have to keep learning on our own.

The final part of this series will be about "analysis." The metrics and dimensions shown in the App Store experiment's analysis results are well-documented in the guide, so I'm planning to write about that weird sense of "frustration" you feel as someone who has actually run the experiments. When testing the same hypothesis on both the App Store and the Google Play Store, the interpretation of the results from the two experiment modules was wildly different. While experimenting with hypotheses to increase app installs, I've also started running experiments just to see how differently the two stores' modules yield results.

Let's take a closer look at those experiments in the next post.

mail : stephan@delightroom.com
instagram : @seosinghan