Paywall Experiments for Subscription Apps: Quick A/B Testing Framework

Growth

Jul 4, 2023

Michael

Paywall Experiments for Subscription Apps: Quick A/B Testing Framework

Growth

Jul 4, 2023

Michael

Paywall Experiments for Subscription Apps: Quick A/B Testing Framework

Growth

Jul 4, 2023

Michael

Paywall Experiments for Subscription Apps: Quick A/B Testing Framework

Growth

Jul 4, 2023

Michael

paywall a/b test
paywall a/b test
paywall a/b test
paywall a/b test

Paywall A/B testing is crucial for unlocking subscription app revenue growth. However, conducting rapid experiments can be challenging, especially when restricted by app release cycles and the workload of the engineering and product teams. In this guide, we discuss the way to avoid these limitations and enable quick and flexible experimentation.

Paywall A/B testing is crucial for unlocking subscription app revenue growth. However, conducting rapid experiments can be challenging, especially when restricted by app release cycles and the workload of the engineering and product teams. In this guide, we discuss the way to avoid these limitations and enable quick and flexible experimentation.

Paywall A/B testing is crucial for unlocking subscription app revenue growth. However, conducting rapid experiments can be challenging, especially when restricted by app release cycles and the workload of the engineering and product teams. In this guide, we discuss the way to avoid these limitations and enable quick and flexible experimentation.

Paywall A/B testing is crucial for unlocking subscription app revenue growth. However, conducting rapid experiments can be challenging, especially when restricted by app release cycles and the workload of the engineering and product teams. In this guide, we discuss the way to avoid these limitations and enable quick and flexible experimentation.

Basics of Paywall A/B Experiments

Basics of Paywall A/B Experiments

When it comes to A/B testing your paywalls, having clear and measurable objectives is key. Think of these objectives as your roadmap—they help you stay on track and ensure you’re working towards something concrete. Try forming your objectives by answering two questions:

  1. What’s my goal for running this A/B test?

  2. What metrics should I track?

What to Track in Your Paywall A/B Test?

Selecting the right metrics is crucial for your paywall experiments. These metrics will help you determine whether the changes you make are effective and align with your overall business objectives. Here are some of the key metrics you should consider:

Selecting the right metrics is crucial for your paywall experiments. These metrics will help you determine whether the changes you make are effective and align with your overall business objectives. Here are some key metrics to consider:

  • Conversion Rate: The percentage of users who move from a free tier to a paid subscription.

  • Average Revenue Per User (ARPU): Insight into how much each user is worth to your business.

  • Churn Rate: The percentage of subscribers who cancel their subscriptions.

  • Trial-to-Paid Conversion Rate: The percentage of users who convert from a free trial to a paid subscription. There are more.

Let's get started with the A/B Test.

1. Create compelling paywall design template

1. Create compelling paywall design template

Your paywall needs to convey the benefits of the premium plan, be aligned with your users onboarding experience, and persuade users to start a trial or subscription. The main visual element, like an image or video, needs to be highly relevant with the content of the app. The text elements can be as simple as a short description of the premium plan or more complex with the detailed bullet points highlighting premium features.

Below is an example of a paywall demonstrating these points, that is simple yet pretty powerful, and that can be used as a template. This example contains the following key elements: 

  1. Appealing graphic aligned with app’s use case

  2. Catchy title with the value proposition

  3. Social proof to reassure users

  4. Clear pricing and free trial offer

  5. Clear Call-to-Action button 

  6. Option to skip the paywall if a user is not ready to subscribe yet

  7. Terms, Privacy, and Restore option as required by App Store guidelines

paywall elements explained

You can create the template based on this layout. Each of the important elements can be controlled, changed, and tested individually to optimize paywall’s performance in the future.

Paywall Template 1:

paywall template with elements

2. Consider adding several paywall templates

2. Consider adding several paywall templates

It’s okay to start with one paywall template as shown above, but as already mentioned, you are likely to need more freedom in experimentation to address all hypotheses your team can come up with. So, you don’t have to limit yourself with one template. You can think through cases where you need to present several payment options to a user, highlight the details of the premium features, or explain how the free trial works. Then you can create several templates with different UI layouts, keeping in mind the main requirements. 

Below we are showing 3 additional popular subscription app paywall layouts. Let’s have a look at them one by one.

Paywall Template 2:

  • Main image 

  • Premium details

  • Two subscriptions/products with different duration/price placed horizontally

  • CTA

Paywall template for Subscription Apps

Paywall Template 3:

  • Main image 

  • Premium details

  • Two subscriptions/products with different duration/price placed vertically

  • CTA

paywall template for subscription app

Paywall Template 4

  • Detailed step-by-step trial explanation

  • One subscription offer

paywall template for subscription app

After creating the templates and adding them to your app, it’s time to decide which template to use, what elements to show, and test any configuration for any of the templates.

3. Configure paywall variants for testing

3. Configure paywall variants for testing

By having several paywall templates as shown above and controlling their elements, you can create a virtually unlimited set of variations to test. Here are just a few of the possible variants for each of the template: 

paywall a/b test variants

From this point onward, you can launch monetization experiments using any tool that you prefer. In the following sections we provide guidance on configuring experiments with these paywalls using Qonversion.

4. Control paywall elements with JSON payload

4. Control paywall elements with JSON payload

Qonversion’s Experiments gives you a simple tool to control the elements on a paywall variations and run experiments quickly. You can set up key-value pairs for every element:

paywall a/b test setup

Qonversion SDK returns the JSON with key-value pairs when you need to show the paywall in the app. For example, below you can see this paywall configured with:

  • Different main image

  • Different social proof element

  • Title variations

  • Different CTA

  • Monthly subscription with higher/lower price point

  • Trial vs. no trial option

Based on the SDK payload a user in group A can be presented with paywal 1A, and a user in group B can see 1B variant or any other required variant.



Experiment #1. Paywall 1A vs. Paywall 1B.

Experiment #1. Paywall 1A vs. Paywall 1B.

paywall a/b test template for subscription app

Experiment #2. Paywall 1B vs. Paywall 1C

Experiment #2. Paywall 1B vs. Paywall 1C

paywall a/b test template for subscription app

Experiment #3. Paywall 1A vs. Paywall 1C.

Experiment #3. Paywall 1A vs. Paywall 1C.

paywall a/b test template for subscription app

5. Launch paywall A/B testing in Qonversion

5. Launch paywall A/B testing in Qonversion

To launch a test, make sure that you have Qonversion SDK installed in your app. Then navigate to the Experiments page in your Qonversion account and select “Create a new experiment”. 

The first step here is to formulate the hypothesis, write it down for your future reference, and choose the primary metric for optimisation.

a/b test hypothesis description

Some of the metrics you can set as primary experiment goal:

  • User-to-trial conversion

  • User-to-paid conversion

  • New trials

  • Trial cancellation rate

  • Trial-to-paid conversion

  • New subscriptions

  • Subscriptions cancellation rate

  • Sales (Revenue in USD)

  • Proceeds (Revenue in USD after deducting app stores’ commission)

  • Refunds

You will be able to see all of the metrics above in the test results, so don’t stress to much on what metric to pick here. 

Next to configure the JSON payload for each experiment variant, follow this experiments documentation.

6. User segmentation and traffic share allocation

6. User segmentation and traffic share allocation

Qonversion allows you to segment users and allocate traffic share for A/B testing. This enables targeting specific user segments and control the distribution of traffic between different variations of your paywall. 

User segmentation and traffic allocation for experiments

You are almost ready to launch. One more thing before starting the experiment is to check that each variant looks as expected. You can easily do it using variant ID in Qonversion.

7. Measure experiment performance in real-time

7. Measure experiment performance in real-time

You do need full visibility into what happens to users after they start a trial or a subscription to be able to make any conclusions in case of subscription monetization model. Qonversion A/B testing module uses Qonversion’s best-in-class precise subscription analytics. So you have the full details of the performance in each experiment group and can make correct conclusions. 

paywall a/b test results

A Quick Recap

A Quick Recap

Let's go through the takeaways of this article:

  1. Paywall A/B testing is all about testing a set of common UI elements.

  2. You can simplify and accelerate the experimentation by having several popular paywall templates implemented and ready to be configured.

  3. With Qonversion, you can configure this unlimited set of paywalls with just a few clicks using JSON payload.

  4. Formulate the hypothesis, setup the targeting and traffic allocation, and you are ready to launch a new Paywall A/B Experiment.

  5. Qonversion provides detailed subscription analytics to see real-time experiment results and make conclusions to drive your app’s revenue growth!