Running a push notification test is hard when you’re not aware of every variable. It’s easy to say that you want to A/B test your content, but can you sit down and list out every element of a push notification that you ought to test?
Let’s say you’re drafting a series of push notifications as part of a new lifecycle campaign. You’ve read about the importance of A/B testing, and you want to optimize your messaging by delivering the content that resonates best with your users.
This is a good idea in theory, but what does it entail? What should you test to optimize your push notifications? Is it better to simply test everything?
To answer those questions, we created a step-by-step guide that lists out every major push notification test you can carry out, along with an example of when you should test it. This is the best way to launch an end-to-end push notification test. If you’re ready, let’s dive in.
Message Copy
The most obvious element of a push notification test is also one of the most important. The copy is the message — it’s the reason why you’re reaching out to your users in the first place. Without well-written, relevant copy, even the most timely message could be regarded as spam.
Consider experimenting with tone and message length in your variants, and running a push notification test to see which ones your audience prefers. It’s also difficult to predict what message length will work best for your users until you test it.
When Should I Use A Push Notification Test?
If you’re developing a new push notification strategy, it could be a good idea to run a push notification test to check your message copy before introducing more dynamic factors, like personalization. This will help you develop best practices that are specific to your audience to guide you through future campaigns.
At the same time, if your personalized messages aren’t providing the results you’d like, it may be a good opportunity to go back to the drawing board and A/B test your copy.
Rich Push Notification Test
Rich push notifications were first introduced in iOS 10 and Android 4.1. Compared to traditional text-only push notifications, rich push messages allow mobile teams to include images, audio, and other media. It’s also possible to customize the message buttons and layout, as shown above.
Rich push presents a big opportunity for companies looking to increase push notification engagement, but it’s not a one-size-fits-all solution. Users don’t always want to load an image or a gif with their push notifications, especially if text would have worked just as well. Deciding when and where to use rich push notifications is a good reason to run a push notification test.
When Should I Test It?
If your push notification engagement is in a slump, try experimenting with a rich push notification test. Adding some variety to your messages could give them an engagement bump.
Even if you already use rich push notifications, you can compare different types of media, like images versus gifs. Rich push virtually guarantees that you’ll grab the user’s attention, so you can iterate on it to find the media that best converts user attention into taps.
Push Notification Test For Audience Segments
Putting aside the push notification content, there’s another factor that’s important to optimize: who you target. Rather than sending messages to all of your app users, you can target your push notifications to audience segments. These segments are groups of users filtered by a given criteria, such as app usage time or in-app events triggered.
It may be tempting to test your push notification content on your whole audience and make generalizations from that, but this approach might not tell you the whole truth. For instance, what if rich push notifications perform well as a whole, but a specific segment of your audience never reads them? You run the risk of losing these users if you keep sending them rich push notifications, even if that strategy works well on average. Doing a push notification testing can help avoid situations like these.
By A/B testing your push notifications on different segments, you can customize the content that each user group experiences, so you’ll always know what works best for your customers.
When Should I Use a Push Notification Test for Audience Segments?
Once you’ve established a few best practices for your overall message content, consider delving into your app usage stats to determine if any segments of your audience respond in a surprising way.
The reason for doing this after settling on your overall message tone and content is because it’s difficult to study audience segments without data. Once you have a body of data that shows that your audience is, on average, responding to what you’re saying, you can run a push notification test to find out if there’s an underserved segment that’s responding poorly.
Message Timing
Timing is another non-content factor that can benefit from a push notification test. It might not sound important, but the data shows that 63 percent of push notifications are sent at the wrong time, while well-timed messages can enjoy 800 percent more opens.
There are two ways to personalize message timing: behavioral triggers and algorithmic scheduling. The former means that users are sent push notifications when they meet given criteria, like not launching the app for seven days or adding an item to their cart without checking out.
As cited above, messages sent with behavioral triggers experience an 800 percent higher open rate, so there’s little doubt that triggered messages are effective. This is where a clever push notification test can really swing the odds.
However, not all messages can be triggered. What if you want to announce a new app feature or a promotion, regardless of the user’s actions? That’s where algorithmic scheduling comes in. Leanplum’s solution to this problem is Optimal Time, a feature that automatically sends push notifications to individual users when they’re most likely to open.
Since Optimal Time is a machine learning algorithm, it goes one step further than simply taking time zones into account — it adjusts its timing on the fly, based on each person’s app usage history. Run a push notification test to find out when your audience responds best, and use what you’ve learnt to improve engagement for future campaigns.
When Should I Test It?
Timing is situational, so a lot depends on the type of campaign you’re running. For a behavioral push notification test, you can try A/B testing slight variants of each trigger. For example, if you’re sending a re-engagement message to activate dormant users, you can start by sending the notification after a week, and then test it against a campaign that sends the message after 10 days.
For non-triggered messages, it may be best to start using algorithmic scheduling right away. Of course, it’s never a good idea to trust a black box, so we always encourage Leanplum customers to A/B test our Optimal Time algorithm. If Optimal Time returns better results, there’s no harm in using it right off the bat.
Message Channel – When To Use A Push Notification Test?
A messaging channel is a means through which you can communicate with your audience. For mobile teams, the most common messaging channels are push notifications, in-app messages, emails, and persistent messages stored in the App Inbox.
The same message delivered through another channel could yield dramatically different results. Consider a discount code that you offer to your most engaged users: since the recipient might not use the code right away, it’s possible that the code will get lost if it’s shared as a push notification or in-app message. That same offer could yield a higher conversion rate if delivered via email or App Inbox, enabling users to refer to it when needed. By running a push notification test, you’ll know exactly which option to go for to ensure you get the best possible results from your offer code.
When Should I Use A Push Notification Test?
The messaging channel fundamentally affects how the content is received, so iterating on it is not as simple as testing minor edits to the copy. Do a push notification test on an alternate channel is when you feel that a broad, strategic change should be made.
In many cases, it’s equally acceptable to send a message through any of the available channels, but one particular channel might resonate with your following. If your message engagement is lagging across all user segments and you can’t identify why, it might be that you’re using a suboptimal channel.
By testing the same message on different channels, you can decide if one of them should be your “default” or primary channel, or if your messages should be shared equally across all available channels.
—
These are the five essential components of a push notification test, but there are several smaller factors within them that can be optimized. What attributes should I use when making an audience segment? Do some events work better as behavioral triggers than others? And what are the differences in rich push notification engagement between iOS and Android?
The good news is that these questions can be answered through the same techniques described in this post. With this in mind, you’re ready to launch an end-to-end push notification test.
—
Leanplum is building the marketing cloud for the mobile era. Our integrated solution delivers meaningful engagement across messaging and the in-app experience. We work with top brands such as Expedia, Tesco, and Lyft. Schedule your personalized demo here.
