At Wattpad, we use Branch links to ensure our A/B testing efforts go beyond the vanity metrics of email open rates, button clicks, and page views. The following are the top three lessons we’ve learned when it comes to A/B testing for app growth.
Lesson #1: Measure Installs Not Click Events When Transitioning Users from Web to App
Moving users from web to app via a smart banner is an acquisition strategy that is familiar to most growth teams. One of the most frustrating parts of A/B testing with smart banners is the limited insight that you receive when testing multiple variations. With most tools, you can only see clicks on banners or measure Google Tag Manager (GTM) download events, which can be highly inaccurate. We’ve found that using Branch helps us solve this key problem while giving us deeper visibility into the installs of our apps. The “full screen takeover” versus smart banner experiment that we conducted provided us with excellent insights into which would lead to more downloads of our app and why.
Which variation leads to more downloads of the Wattpad app?
Our control full screen takeover verses our variation smart banner.
Our control, the full screen takeover, received the highest total number of clicks on the call to action versus our “smart banner” variation. However, we knew this was a vanity win, and not reflective of actual installs. To make sure we were measuring actual installs, we appended Branch URL unique parameters to our variation URLs. Through Branch, we were able to see that the smart banner variation had improved actual installs and opens by +1.62%. This may seem like a small win, but with millions of visitors to the Wattpad site, it has a significant impact.
Our hypothesis was that people clicked on our full screen takeover control in an attempt to dismiss it, rather than with an intent to download. So even though our smart banner variation lost in clicks, it won in installs and re-opens, which are the metrics that help dictate sustainable and accurate growth. We now always use Branch to track the installs of every iteration of our smart banner.
Update: A/B testing for Branch smart banners is now a built-in part of our Journeys web to app product.
Lesson # 2 Optimize Emails For The Right Metric
Our growth roadmap includes some pretty aggressive goals for growing our monthly active user base. To do so, our strategy involves experimenting with both acquisition of new users, like in our smart banner above, and retention-focused experiments for existing users.
We already know our app users are retained twice as well as our web users. Our smart banner is one way to migrate website signups into well retained app users.
Testing Email Subject Lines for App Conversions
With the help of Wattpad’s Data Insights team, we identified a substantial set of web and mobile web signups who had both never downloaded the Wattpad app and who had opted in to receive emails from us. We created a large-scale “Download the App” email campaign where we tested two different subject lines, one of which was click-bait. Our email marketing tool lets us track email opens and clicks, but we didn’t want to optimize just for clickbait. So we leveraged Branch to see which variation resulted in the most app installs.
By using campaign-specific Branch links, we were able to see which of our emails converted web users into the better-retained app users. Our winning email improved conversion rates by 150.08%. Surprisingly, the more clickbait-y subject line resulted in a worse conversion rate and fewer overall installs. Our hypothesis is that a realistic subject line better conveyed the purpose of the email, and thus drove users further down the conversion funnel.
Update: Branch now offers a complete deep linking solution for emails. Learn more to see which email service providers we support.
Testing App Downloads and Bounce Rates from Email
Using Branch links also just makes sense; sending mobile users to the correct app store or directly to your app from the click of a button beats the experience of opening up a mobile web browser any day. In a separate email experiment, we replaced all links in an existing email with Branch links. In the email without the Branch links, we saw a bounce rate of 62.76% and fewer overall downloads using GTM events. In the Branch version of the emails, we saw a 5.8% conversion rate to download and a 25.4% conversion to app opens. That means by using Branch, we saved almost a third of our users from an annoying mobile web experience, where they have to go multiple steps to open content directly in their app. The deep linked emails improved our downloads by 109%, improved our session time by 96%, and lowered our bounce rate by 27%.
In order to encourage users to download our app, we redirected our desktop users to our Get Mobile site where they can text themselves a link to download the app, rather than be sent to an app store on desktop. With help from Branch, we were able to create a seamless conversion experience for mobile web and desktop users, which boosted our confidence in our ability to scale our email performance.
Lesson #3: Let Users Choose Which Ad Creative Converts Best
Three variations of deep linked app ads
While working with our Business Development team, we realized we had an opportunity to test serving internal house ads on our unsold ads inventory. We’ve experimented in the past with Facebook ads, but testing multiple versions of an ad can get expensive and we wanted to find out which creative works best on web and mobile web.
We didn’t want to depend on vanity metrics like whether users were clicking on the ad, we wanted full visibility into a user’s journey from click to download, including their device type. We created three versions of the ads with different imagery, as well as text inspired by what we’ve tested elsewhere.
Branch tracking gave us insight into which ad worked best for both opens and installs. Our ad featuring mobile devices (variation #3) converted users much more effectively, improving app conversion by up to 142.12%. Visibility into which ads performed the best (our mobile web ads) and where we were seeing the most installs (Android) also improved.
A/B testing and experimentation is the backbone of Wattpad’s growth team. Before Branch, we were often dependant on vanity metrics like clicks that could be indicative of attempts to dismiss a banner, rather than intention to download our app. Getting full visibility into organic attribution, install, and in-app metrics now means we can test better and faster, and hold ourselves accountable to metrics that actually matter for our growth. We’re in the process of expanding our use of Branch on the growth team and in the product overall, and we’re excited about our additional upcoming experiments.
Want to start A/B testing your app emails, ads, banners, and more? Use the button below to get started.
Special thanks to Emily Nguyen and Zoe Di Novi for contributing this blog.
Emily manages all things acquisition at Wattpad. She’s passionate about the intersection of data, product and marketing to achieve measurable results. Follow her on Twitter @nguyene.
Zoe Di Novi is the Product Growth Lead at Wattpad, and is super proud of the team’s data-driven and experimentation-oriented approach to growing MAU. Follow her on Twitter @zoedinovi.