A/B Paywall Experiments for Subscription Apps Step-By-Step Guide 

Paywall A/B experiments are crucial for successful subscription app monetization. Qonversion’s A/B Experiments feature provides everything required to quickly launch experiments, analyze results, and roll out winning versions without releasing a new app build.

Qonversion A/B Experiments include:

• User segmentation by country, install date, app version, free/paying user
• Traffic allocation
• Subscription analytics including trial-to-paid conversion rate, subscription renewals, total revenue, cancellation rate, and more
• Visualization of A/B experiments performance 
• Statistical significance of the results (no need for external statistical calculators)
• Roll out winning versions without app release (Remote Config)

To run A/B experiments with Qonversion sign up and install Qonversion SDK.

1. Formulate Experiment Hypothesis

Start with formulating a hypothesis for the experiment. For example, let’s assume the following hypothesis:

• Price increase in the US: By increasing the monthly subscription price by 25% for US users and changing the main picture to more visually appealing we can increase subscription revenue by 20%+.

2. Get ready to experiment: configure your app monetization details to be managed remotely

This is where the tech team needs to be involved to start your first experiment.

Qonversion’s Remote Config is a feature that helps you remotely change the structure/design of your app paywall without re-releasing your app. This helps you make quick changes to your paywalls targeted for specific groups of your users and iterate quickly with your experiments (speed of experimentation is crucial for rapid app growth).

So, before starting to experiment let’s link your paywall elements with Qonversion’s Remote Config. 

1. Navigate to ‘Remote Config’ in your Qonversion account and create a new Remote Config.

Click ‘Edit’ payload and paste key-value parameters that refer to the elements of your paywall. As we’d like to test images and prices, let’s add these parameters. However, you can link all the paywall elements, including text, buttons, CTAs, value propositions, ratings, etc.

Once you fill in the default paywall details in the dashboard, you can get these elements inside the app. 

Call the Qonversion remoteConfig method and use identifiers from the dashboard.

Qonversion.shared().remoteConfig { remoteConfig, error in
    // Image URL from Qonversion Remote Config
    let imageUrl = remoteConfig?.payload?["imageURL"] ?? "default_image_url"
    // Product ID from Qonversion Remote Config
    let topProductId = remoteConfig?.payload?["topProductId"] ?? "sub.month.trial.3.99"
}

Then, test your default paywall before clicking ‘create’. It will help you to test the setup without rolling the changes for the entire scope of users.

Pass the values to Qonversion SDK by using the attachUserToRemoteConfiguration method.

Qonversion.shared().attachUser(toRemoteConfiguration: "your_remote_configuration_id") { success, error in
     // handle result
}

In case your user has already been attached to another configuration, use the detachUserFromRemoteConfiguration method.

Qonversion.shared().detachUser(fromRemoteConfiguration: "attached_remote_configuration_id") { success, error in  
    // handle result  
}

You can always check the source of the Remote Config you received. Use the remoteConfig.source.type field. Potential values you can receive:

• .experimentControlGroup
• .experimentTreatmentGroup
• .remoteConfiguration
• .unknown

Additionally, you can check whether Remote Config was assigned by Qonversion or by yourself. Use the remoteConfig.source.assignmentType field. Potential values.

• .auto
• .manual
• .unknown

 Now, return to the dashboard and click ‘Create”.

Great! Thus, we installed the paywall to be managed remotely. Now you can make any changes to the paywall and roll it out to a specific audience without additional app releases. In the next part, we’ll show you how to set up A/B experiments for your subscription apps without needing to rerelease your app.

3. Create a new experiment

In the previous section, we installed the default paywall. As we described above, we’d like to test other variation of this paywall for a specific audience – the US. Just as a reminder – we need to compare this default paywall with the paywall with the updated image and price. 

To create an experiment based on your default paywall as a control group, go to the list of your Remote configs, choose the proper paywall, and click ‘Create experiment’. 

Then, add your hypothesis statement. Assign your own name and hypothesis for your experiment.

4. Choose a subscription metric for optimization

Select the main metric targeted in the experiment from the dropdown list. In our example, we selected total revenue (sales).

5. Configure experiment variants

This is where your developer needs to be involved as well. 

Navigate to the Configure tab. 

The first magic that you’ll get is that you already have a control group installed. As mentioned above, this happened because you created your experiment based on your default remote config. 

Next, copy your control group and change values in the existing attributes.  

Change the name of your new paywall.

Then, change the key-value parameters based on your new paywall idea. 

As we have already configured Remote Config, all you need to do is pass the new variable’s parameters to your app. However, it is necessary to test whether your new paywall works properly. Similar to the previous steps, we have a few methods to check this. Do not ignore this part because testing is crucial to ensuring a smooth experience for your users.

Copy the key for your experiment and the variant you want to test.

Pass the values to Qonversion SDK by using the attachUser method.

Qonversion.shared().attachUser(toExperiment: "your_experiment_id", groupId: "your_group_id") { success, error in
     // handle result
}

In case your user has already been attached to another experiment, use the detachUser method.

Qonversion.shared().detachUser(fromExperiment: "attached_experiment_id") { success, error in  
    // handle result  
}

You can always check the source of the Remote Config you received. Use the remoteConfig.source.type field. Potential values you can receive:

• .experimentControlGroup
• .experimentTreatmentGroup
• .remoteConfiguration
• .unknown

Additionally, you can check whether Remote Config was assigned by Qonversion or by yourself. Use the remoteConfig.source.assignmentType field. Potential values.

• .auto
• .manual
• .unknown

6. Segment the audience

Return to the dashboard. Set the experiment segmentation to target only relevant users. In our example we target only US users. In the segmentation section click add rule and select appropriate segments: 

As you will see, you can target your experiment for a very specific audience based on criteria such as country, app install date, app version, store, and users’ active subscriptions. In this case, we’ll roll out the experiment for the segment of new users (who installed the app after the 1st of September) from the US.

You can see all applied segments, make sure that it is aligned with your hypothesis.

7. Set traffic allocation

You can start the experiment with just a fraction of your traffic. For example, roll out the experiment to 20% of eligible users:

8. Start the Experiment and Monitor Performance

Start the experiment and navigate to the “Analyse” tab to monitor experiment performance.

You have the data for the metric you targeted with the experiment at the top of your screen with details on statistical significance of the current values on the right. The data is shown in real time. 

Avoid stopping the experiment until it has reached statistical significance. Also, it make sense to stop experiment if the reaching the statistical significance is not possible.

You have the data for the metric you targeted with the experiment at the top of your screen with details on statistical significance of the current values on the right. The data is shown in real time. 

Avoid stopping the experiment until it has reached statistical significance. Also, it make sense to stop experiment if the reaching the statistical significance is not possible.

You can see how the metric evolves over time on a chart. You can switch between metrics to understand performance of the experiment. The data for control and experiment groups in full details is also available in the table below the chart:

9. End the experiment

Once you have identified the winning variation with statistical significance you can end the experiment.

If you want the data on subscription renewals and refunds to continue to be added to the experiment’s results do not stop the experiment, click ‘pause’. In this case, the users already in the experiment stay in the experiment and the subsequent revenue events are being added. You can measure the retention and continue getting the data on total revenue in both groups.

If you want to be able to expose the users from this experiment to new experiments, you can stop the experiment. In this case, the subsequent revenue events like subscription renewals won’t be added to the experiment results after you stop it.

10. Roll out the winning variant with remote config

Once you completed the experiment, you can easily roll out your winning variation without app release using remote config. Once you click “End experiment”, you’ll see a notification “Create Remote Config?”. Click ‘Create”. 

You’ll be automatically redirected to the Remote Config settings with the parameters of your winning variant. 

Please note that the new Remote Config doesn’t replace your existing one. So deactivate or adjust the old one, if necessary. For instance, in our case, we aim to increase the price just for the US; all other countries should see the default paywall. 

That’s why we need to go to the settings of our default paywall and exclude the United States of America from the target list so that they see the correct paywall that we just rolled out.

That’s it! 

Conclusions

Qonversion’s A/B Experiments enable seamless and flexible experimentation for subscription apps from hypothesis formulation to audience segmentation and result interpretation – and all this without app release. The ability to roll out paywall changes without requiring an app release adds a layer of convenience and efficiency to the entire process. You do not need to juggle multiple tools or second-guess your decisions. The accuracy and statistical foundation backing Qonversion’s experiments ensure reliable insights to empower your app’s growth. Try now.