A Step-By-Step Guide to Set Up Paywall A/B Tests in Qonversion

Kate

Oct 20, 2023

We’ve already explored the fundamentals of A/B testing and identified key elements to test within a paywall. Now, it's time to dive into the practical aspect: setting up Paywall A/B Experiments in Qonversion. In this comprehensive guide, we will walk you through each step of the process, ensuring you can efficiently set up, manage, and analyze your A/B tests to optimize your app’s subscription conversions. Whether you’re new to app monetization or already got your hands in this, our guide will help you leverage Qonversion’s robust A/B testing capabilities.

A/B Paywall Experiments in Qonversion

1. Set Up Remote Config for Your Paywall

To kick off your first A/B test, you'll need to prepare your app for experimentation. This involves configuring your app's monetization details so they can be managed remotely—a task that requires collaboration with your tech team.

Qonversion’s Remote Config as the name suggests allows you to remotely change the structure and design of your app paywall without releasing a new app version. Here’s why this matters: the speed of experimentation is critical for rapid app growth. By using Qonversion’s Remote Config, you can:

Quickly Implement Changes: You can adjust pricing, visuals, play with Qonversion Experiments on the fly without waiting for app store approvals.
Target Specific User Segments: You can tailor your paywalls to different groups based on demographics, usage patterns, and more.

So, before starting to experiment, let’s link your paywall elements to Qonversion’s Remote Config.

Navigate to Remote Config in your Qonversion account and create a new configuration by adding the Configuration name and Context key.
Once you create a configuration, you’ll see it in the list.

Click Edit payload and paste key-value parameters that refer to the elements of your paywall.

Let’s add these parameters to test images and prices. You can link all the paywall elements, including text, buttons, CTAs, value propositions, ratings, and whatever you need.

You can get these settings inside the app once you fill in the default paywall details in the dashboard.

Call the Qonversion’s remoteConfig method and use identifiers from the dashboard.

Qonversion.shared().remoteConfig(contextKey: "my_context_key") { remoteConfig, error in // Image URL from Qonversion Remote Config let imageUrl = remoteConfig?.payload?["imageURL"] ?? "default_image_url" // Product ID from Qonversion Remote Config let topProductId = remoteConfig?.payload?["topProductId"] ?? "sub.month.trial.3.99" }

Then, test your default paywall before clicking Activate. This will help you test the setup without rolling out the changes to all users. By default, Qonversion chooses which configuration to return to the user based on the running options. This method helps to validate how these settings in Draft operate.

Copy your configuration ID and pass the values to Qonversion SDK by using the attachUserToRemoteConfiguration method.

Qonversion.shared().attachUser(toRemoteConfiguration: "your_remote_configuration_id") { success, error in // handle result }

If your user has already been attached to another configuration, use the detachUserFromRemoteConfiguration method.

Qonversion.shared().detachUser(fromRemoteConfiguration: "attached_remote_configuration_id") { success, error in // handle result }

Test before going live.

You can always check the source of the remote configuration you received. By using the remoteConfig.source.type field, you’ll receive one of these:

.experimentControlGroup
.experimentTreatmentGroup
.remoteConfiguration
.unknown

Additionally, you can check whether the remote configuration was assigned by Qonversion or by you. Use the remoteConfig.source.assignmentType field. Potential values:

.auto
.manual
.unknown9

Now, return to the dashboard and click Аctivate on the right upper corner.

Congrats! You have now successfully installed the paywall to manage it remotely. Now, you can make any changes to the paywall and deploy them to specific audiences without requiring additional app releases. We'll create your A/B test in Qonversion Experiment based on this configuration, it's going to be a lot easier, let's go!

2. Formulate a Hypothesis for Your A/B Test

Now that your Remote Configs are set up let’s dive into setting up Qonversion’s Experiments — essentially the best A/B tests for your subscription apps. Before we get to experiment creation, you need to formulate a hypothesis. For this experiment, we’ll take this:

By increasing the monthly subscription price by 25% for US users and changing the main picture to a more visually appealing one, we can increase subscription revenue by 20%+.

In case you’re lost, look for hypothesis ideas here. Once your Remote Configs are set up and you know what you’re going to test, go ahead and:

Then, add your hypothesis statement. Assign your own name and hypothesis for your experiment.

3. Create an A/B Experiment

We create a remote configuration to define the default app behavior. You can (and you should) launch an A/B test based on a remote config. This can work as a benchmark in your A/B test. So basically, the Remote Config we created earlier will be your Control Group for this A/B test.

To create an A/B test, navigate to the Remote Config, select the needed configuration, and click Create Experiment.

4. Plan Your App Optimization Metrics

Select the primary metric you will test in this A/B experiment based on your hypothesis. After analyzing numerous apps and A/B tests for subscription apps, we suggest an extensive list of metrics that include the ability to track user-to-trial conversion, user-to-paid conversion, new trials, trial-to-paid conversion, sales, cancellation rates, proceeds, refunds, and more.

key metrics to track in paywall A/B test

5. Set Up Variant Groups for Your A/B Test

Navigate to the Configure Tab in your Qonversion dashboard. You'll notice that a control group has already been installed. This happens automatically because your experiment is based on your default remote configuration.

paywall A/B test control vs variant groups

To create a variant group, simply copy your control group and modify the values of the existing attributes. Go ahead and change the name of your new paywall and adjust the key-value parameters according to your new paywall idea.

Since the Remote Config is already set up, you don’t need to pass the new variable parameters to your app: earlier, you’ve configured the default paywall with this code:

Qonversion.shared().remoteConfig(contextKey: "my_context_key") { remoteConfig, error in // Image URL from Qonversion Remote Config let imageUrl = remoteConfig?.payload?["imageURL"] ?? "default_image_url" // Product ID from Qonversion Remote Config let topProductId = remoteConfig?.payload?["topProductId"] ?? "sub.month.trial.3.99" }

The new configuration for the the experimental paywall will appear here automatically—you don’t have to modify the code.

What you have to do though, is to test the new paywall and ensure it works correctly. To check this, Copy the Key for Your Experiment and Variant.

Use the following method to attach a user to your experiment:

Qonversion.shared().attachUser(toExperiment: "your_experiment_id", groupId: "your_group_id") { success, error in // handle result }

If a user is already attached to another experiment, detach them using:

Qonversion.shared().detachUser(fromExperiment: "attached_experiment_id") { success, error in // handle result }

Once again, we remind you that you can test your remote configuration. To verify the source of the Remote Config use the remoteConfig.source.type field. Potential values include:

.experimentControlGroup .experimentTreatmentGroup .remoteConfiguration .unknown

To determine assignment type check whether the Remote Config was assigned by Qonversion or manually using the remoteConfig.source.assignmentType field. Potential values are:

.auto .manual .unknown

Testing each configuration is critical to ensure a smooth user experience. Do not skip this step and validate the functionality and performance of your new paywall settings.

6. Segment the Audience of your A/B Experiment

By carefully segmenting your audience, you can target only the relevant users to ensure that your A/B test results are accurate and relevant. Go ahead and Add Rule(s) to define the criteria for your target audience.

You can specify the audience based on a list of criteria, such as:

Country
App Install Date
App Version
Store
Users’ Active Subscriptions
Attached experiment context key
Attached active experiment

Use the last two in case you have the same users in multiple experiments to ensure a smooth user experience and clear A/B test results.

Once you have added the rules, review all applied segments to ensure they align with your hypothesis and move on to the next step.

7. Allocate Traffic to Your A/B Experiment

Think of how many people need to see your A/B test so that you have significant results but don’t jeopardize the experience of all of your users. In Qonversion, you can start the experiment with just a fraction of your traffic to test the changes on a smaller scale before a full rollout.

8. Start the Experiment

When all is set and done, you’ve tested the app behavior through Remote Config, and you sure the A/B test doesn’t mess up your user flow, go ahead and click Start.

What’s important to note here is that you need to keep the experiment running until you reach a statistical significance, which means until you see any notable results not connected to chance, timing, and other inconsistent factors. And the vice versa, it make no sense to run an experiment when it’s impossible to reach the statistical significance, e.g. the groups are too small, the timing is ineffective, etc.

9. Analyse the Results of A/B Test

When your experiment is running, navigate to the Аnalyse tab to monitor its performance. This section provides real-time data for the targeted metrics, along with details on the statistical significance of the results.

Key points to focus on include:

Primary Metric Overview: The top of the screen displays data for your primary metric, including real-time updates and statistical significance details.
Progress Monitoring: Use the chart to track how your metrics evolve over time. You can switch between different metrics to get a comprehensive understanding of the experiment’s performance.
Detailed Comparison: Below the chart, a table provides full details on the control and experiment groups.

10. End the Experiment

Once you've identified the winning variant with statistical significance, you can end the experiment in two ways:

Pause vs. Stop: If you want to continue collecting data on subscription renewals and refunds, choose pausing your A/B test. This ensures that users already in the experiment remain in it, and subsequent revenue events continue to be added.
Stop Experiment: If you prefer to expose users from this experiment to new experiments, stop the experiment. Note that subsequent revenue events like subscription renewals won’t be added to the experiment results after stopping.

Now you may ask, why pause experiments?

Pause the Experiment

Sometimes, you might want to halt the exposure of new users to the A/B test without ending the experiment entirely. Why? Well, for subscription apps with recurring revenue, paywall changes directly affect your revenue. You may want to see how specific sales affect conversion, but that doesn’t mean you want to keep showing this sale to new users monthly. In this case, pausing the experiment gives you:

Continued Data Collection: Data continues to be sent to your app for users already involved in the test.
Zero Traffic Allocation: The traffic allocation percentage is zero, ensuring no new users are affected.

This approach allows you to analyze exposed users thoroughly while preserving the integrity of ongoing data collection.

Rolling Out the Winning Version

After completing the experiment, rolling out the winning variant is straightforward with remote config. Here’s how:

End Experiment
Once you click End Experiment you’ll see a notification asking if you want to ‘Create Remote Config?’. Click Create.
Remote Config Settings
You’ll be redirected to the Remote Config settings with the parameters of your winning variant.
Adjust and Deploy
Review and adjust the data as necessary. For example, to apply new pricing to all users, you might need to remove certain segmentation criteria, such as app install date.

Remember, the new Remote Config doesn’t replace your existing remote configuration. Deactivate or adjust the old one to display the correct paywall to all users.

Wrapping up A/B testing in Qonversion

Qonversion’s A/B Experiments offer a seamless and flexible approach to experimenting with subscription apps. The entire process is streamlined and efficient, from hypothesis formulation to audience segmentation and result interpretation. Rolling out paywall changes without requiring an app release adds a significant layer of convenience. You can confidently rely on the statistical foundation of Qonversion’s Experiments to gain reliable insights and drive your app’s growth. Talk to us today to unlock the full potential of your subscription app!

Growth

A Step-By-Step Guide to Set Up Paywall A/B Tests in Qonversion

Kate

Oct 20, 2023

We’ve already explored the fundamentals of A/B testing and identified key elements to test within a paywall. Now, it's time to dive into the practical aspect: setting up Paywall A/B Experiments in Qonversion. In this comprehensive guide, we will walk you through each step of the process, ensuring you can efficiently set up, manage, and analyze your A/B tests to optimize your app’s subscription conversions. Whether you’re new to app monetization or already got your hands in this, our guide will help you leverage Qonversion’s robust A/B testing capabilities.

A/B Paywall Experiments in Qonversion

1. Set Up Remote Config for Your Paywall

To kick off your first A/B test, you'll need to prepare your app for experimentation. This involves configuring your app's monetization details so they can be managed remotely—a task that requires collaboration with your tech team.

Qonversion’s Remote Config as the name suggests allows you to remotely change the structure and design of your app paywall without releasing a new app version. Here’s why this matters: the speed of experimentation is critical for rapid app growth. By using Qonversion’s Remote Config, you can:

Quickly Implement Changes: You can adjust pricing, visuals, play with Qonversion Experiments on the fly without waiting for app store approvals.
Target Specific User Segments: You can tailor your paywalls to different groups based on demographics, usage patterns, and more.

So, before starting to experiment, let’s link your paywall elements to Qonversion’s Remote Config.

Navigate to Remote Config in your Qonversion account and create a new configuration by adding the Configuration name and Context key.
Once you create a configuration, you’ll see it in the list.

Click Edit payload and paste key-value parameters that refer to the elements of your paywall.

Let’s add these parameters to test images and prices. You can link all the paywall elements, including text, buttons, CTAs, value propositions, ratings, and whatever you need.

You can get these settings inside the app once you fill in the default paywall details in the dashboard.

Call the Qonversion’s remoteConfig method and use identifiers from the dashboard.

Qonversion.shared().remoteConfig(contextKey: "my_context_key") { remoteConfig, error in // Image URL from Qonversion Remote Config let imageUrl = remoteConfig?.payload?["imageURL"] ?? "default_image_url" // Product ID from Qonversion Remote Config let topProductId = remoteConfig?.payload?["topProductId"] ?? "sub.month.trial.3.99" }

Then, test your default paywall before clicking Activate. This will help you test the setup without rolling out the changes to all users. By default, Qonversion chooses which configuration to return to the user based on the running options. This method helps to validate how these settings in Draft operate.

Copy your configuration ID and pass the values to Qonversion SDK by using the attachUserToRemoteConfiguration method.

Qonversion.shared().attachUser(toRemoteConfiguration: "your_remote_configuration_id") { success, error in // handle result }

If your user has already been attached to another configuration, use the detachUserFromRemoteConfiguration method.

Qonversion.shared().detachUser(fromRemoteConfiguration: "attached_remote_configuration_id") { success, error in // handle result }

Test before going live.

You can always check the source of the remote configuration you received. By using the remoteConfig.source.type field, you’ll receive one of these:

.experimentControlGroup
.experimentTreatmentGroup
.remoteConfiguration
.unknown

Additionally, you can check whether the remote configuration was assigned by Qonversion or by you. Use the remoteConfig.source.assignmentType field. Potential values:

.auto
.manual
.unknown9