Hypothesis Testing with Braze (1)

Key Braze Limitations You Should Know

2

This series is a "Braze journey" that shares the key limitations first-time Braze users will face and explores how we overcame them to conduct meaningful hypothesis testing. In this first part, I've rounded up the key limitations that those in charge should keep in mind.

When I joined DelightRoom, my first impression of Braze was: "An absolute must-have tool for any org thirsty for growth!" Of course, I thought that without knowing it was so expensive... But as long as the integration is done right, it’s a tool that lets you independently experiment with various hypotheses without any extra client-side local engineering work. You can send push messages and in-app messages in multiple formats to users in the app, and you can use email too. You can also customize the message copy by pulling in user properties, and above all, you can configure the message trigger criteria and target segments to be incredibly sharp.

My second impression of Braze was: "A tool that’s even more charming because it’s a bit crude and clunky!" Of course, I didn't know that clunkiness would lead to major operational inefficiencies and headaches... But the fact that Braze, as a startup's product itself, was constantly changing and growing was quite appealing. Even the features that hadn't been released or were only half-released felt like very high-impact product specs when looking at their direction. It felt like a tool with a lot to look forward to.

But that excitement was fleeting. When I actually sat down at my desk to prepare various hypotheses and use Braze for real, tough hurdles stood in my way. Since I hadn't mapped out the product specs with an A-to-Z understanding of its functional limitations, I had to pivot the planning direction mid-way or drop ideas entirely depending on the constraints I encountered during setup. I've outlined a few of the major limitations below, along with real-world examples.

Hurdle 1: Can't use Canvas

The biggest surprise(?) was that we couldn't use Canvas, the core feature they heavily pitched and emphasized. Canvas is a feature that lets you map out a user's in-app messaging journey all at once, making it efficient to set up and manage multiple campaigns. "If a user who gets the first message does this action (clicks the CTA or closes it), they get this next message depending on each action, and based on their action to the follow-up message, we differentiate the next message again..."

Ironically, this highly appealing feature was unavailable for "in-app messages." The messaging channels you can use through Braze are "push messages," "in-app messages," and "email." Among these, Alarmy was getting its main utility from "in-app messages," but there was a fatal limitation that in-app messages couldn't be mapped out in Canvas.

Also, within Canvas, the use of action-based triggers was restricted. Braze lets you send messages at a specific time or use a user's specific behavior (action) as a trigger. For in-app messages in particular, action triggers are more effective than for push messages or emails (since they can be triggered after a specific action while using the app, making it look like a naturally flowing screen). However, in Canvas, you can only use action-based triggers in the very first campaign of the sequence; for all subsequent flows, only time-based triggers are allowed. For example, for users who clicked and opened the first push message in the sequence, you can't send a second push message triggered by their action of dismissing an alarm. You have to set it based on time so that the message goes out around 9 AM the next morning. It essentially means we can't deliver messages with the sharp, impactful precision we want.

So, we gave up on using Canvas. We had no choice but to painstakingly stitch together the user journey campaign by individual campaign.

Hurdle 2: Can't trigger messages based on server event values

Braze has events and user properties that they provide out of the box, as well as custom events and custom user properties that clients build and utilize themselves. Naturally, as the name implies, custom values are much more useful (and precise) when customized for message triggers and segment targeting. Things like firing a message to a user who clicked a specific button in the app or a user who has never visited a specific screen need to be based on custom values.

But among the collected events, you can't trigger messages using server event values. For instance, I wanted to fire a message at the exact moment someone canceled their subscription, but the server event where the user's subscription status value changes couldn't be used as a trigger. To use it as a trigger, we had to plant a client event on the cancel subscription button, separate from the server event, and use that as the trigger. Alternatively, since you can segment (target) via server events, targeting users who have ever triggered that event is another method. We couldn't fire it at the exact moment the event occurred, which blunted the timing of the messaging, but we created a segment of people who had canceled their subscription within the last day so they would receive a message when they launched the app.

• Trigger messages based on server event values [Impossible]

• Trigger messages based on client event values that affect server event value changes [Possible]

• Create user segments based on server event values and trigger messages when they open the app [Possible]

Hurdle 3: Can't create user segments based on specific event properties

This time, I ran into a hurdle with user segments (targeting). I wanted to target users who had never visited a specific screen, but I couldn't. Because using event properties when creating user segments is impossible.

If there are 4 screens in the app and you are collecting each as separate event values like Pageview A, Pageview B ... Pageview D, you can segment based on whether they visited that screen. You just set it to things like "Has Pageview A more than 0 times."

However, if the tracking is designed so that the Pageview events for all 4 screens are collected as a single event and identified by a property value like page_type = A, B, C, D, it is impossible to segment out only the users who visited screen A. You end up with a blunt targeting rule like "Has Pageview more than 0 times."

By event design standards, it is common to identify events via property values beyond a certain depth, so this felt like a critical limitation. Also, even if the depth isn't deep, you might just want to target users based on a specific event property. For example, our "click purchase button" event logs the selected product as a property. It could be an "annual subscription" or a "monthly subscription." Even if I only wanted to target users who selected "monthly subscription" and clicked purchase, I currently can't do that in Braze.

At this point, I discovered that while I couldn't create a user segment using a specific event's properties, I could set it as a message trigger, so I managed to scrape together a weird workaround(?) to test the hypothesis I wanted. I can't target only the users who selected "monthly subscription" and clicked purchase, but I can send a message at the moment they select "monthly subscription" and click purchase. Originally, I wanted to target the users who selected "monthly subscription" and clicked purchase and send a message triggered by them re-entering the purchase screen, but since that targeting wasn't possible, I had to sacrifice the sharp messaging timing to supplement the targeting. I compromised with myself by setting the message to be sent 30 seconds after the action of selecting "monthly subscription" and clicking purchase.

• Target users who have clicked to purchase a monthly subscription and fire a message at a specific time [Impossible]

• Set the message to send n seconds after the action of clicking to purchase a monthly subscription [Possible]

Hurdle 4: Hard to create user segments

What was interesting in the targeting section was that they provided a variety of user targeting templates. But to achieve the targeting I wanted using the standard templates Braze offered, I had to cobble things together by mixing and matching this and that.

For instance, they don't provide a template to target "day n since starting a free trial." So I had to make do with a combination of templates: "More than n-1 days since starting free trial" and "Not more than n days since starting free trial." (Image below)

It felt a bit like, "Complete the desired essay using only these provided sentences."

Even when targeting based on the same "action," there are multiple types of templates provided. (Image below)

• How long has it been since they first did a specific action?

• How many times have they done a specific action?

• Have they done a specific action within the last n days?

Assuming it's a great service, maybe I could define and express every target group I want using the templates Braze provides. However, based on the shortcomings I've already discovered (Hurdles 1-3 above), I have a strong feeling that combinations of segment templates won't be able to cover every use case. (I really wish they'd fix Hurdle 3 first. For Hurdle 4, I'm sort of finding workarounds...)

Overcoming these hurdles, I finally started experimenting with a few hypotheses. This was even at a time when custom events and user properties weren't planted in granular detail yet. The targeting and triggers weren't as sharp as I wanted, but I pushed forward with the first experiment, adopting a mindset of taking it one step at a time through trial and error.

I wanted to target users who hadn't visited the premium feature screens, but that failed due to Hurdle 3, and I wanted to target users who had selected a specific product, but that also failed due to Hurdle 3. I wanted to differentiate messages based on user property values, but we weren't collecting property values within Braze, and I wanted to land users on a specific feature, but deep linking hadn't been set up yet.

Thus, our first experiment ended up being a messaging experiment that recommended a premium free trial to new users 5 minutes after their first app launch. Stripping away a lot of product spec complexity and keeping it very simple, we basically started with an experiment of firing a message to all new users.

[To be continued in the next part]

mail : stephan@delightroom.com
instagram : @seosinghan