Running experiments
Test Flows against each other, or test variants inside a Flow, at one Checkpoint or across several.
Targeting rules decide which customers reach the Flow (or Offering) served by that rule at a Checkpoint. Experiments test different Flows, Offerings, or steps against each other for those customers, so you can see what performs best.
Experiments only enroll customers who reach them through a Targeting rule. You control which Checkpoints serve each experiment, and the same audience can encounter multiple experiments throughout their lifecycle.
For example, the same Audience could encounter:
- An onboarding Checkpoint testing different introductory offers
- A feature gate Checkpoint testing different upgrade messages
- A home screen Checkpoint testing different Paywall designs
These experiments each run independently, and are measured independently.
To run an experiment at one Checkpoint only, run it from a rule at that Checkpoint. To share it across Checkpoints, add each Checkpoint to the same Targeting rule.
Choose an experiment type
There are two ways to run an experiment at a Checkpoint:
| Type | Variants | Use it to |
|---|---|---|
| Between Flows | Multiple published Flows or Offerings | Compare complete experiences, like two Paywalls you've already built. |
| Inside a Flow | Multiple branches of steps within the same Flow | Test one part of a longer Flow, like a single screen in an onboarding Flow. |
Run an experiment between Flows
An experiment between Flows splits the customers who match a Targeting rule across multiple Flows or Offerings. You set it up from the rule itself.
Before you start:
- Publish each Flow you want to test. Only published Flows can be variants. Web Funnels and Flows that contain their own experiment can't be variants.
To set up the experiment:
- In Targeting, open the rule, or click New rule to create one.
- In Targeting rule setup, select the Flow card, then click Run experiment.
- The Flow or Offering the rule served becomes variant A (Control). If the rule's Flow was never published, RevenueCat publishes it first.
- If the rule didn't serve anything yet, every variant starts empty.
- In Experiment settings, choose a Flow or an Offering for each variant from Select flow or offering. Click Add variant to test more variants.
- Set the Name and the Primary metric. The default metric is Initial conversion rate.
- Under Audience, set the Percentage of customers to enroll. The default is 100%.
- Click Start, then click Make live to turn the rule on. Customers only reach the experiment while the rule is live.
While the experiment runs, its variants, name, and primary metric are locked. You can still change the enrollment percentage.
Variant Flows stay editable while the experiment runs or is paused. Publishing changes to a variant Flow updates that variant for every enrolled customer, which affects your results. You can't unpublish or delete a Flow while it's a variant.
Run an experiment inside a Flow
An experiment inside a Flow splits customers across multiple branches of steps in the same Flow. Use it when the variants share the rest of the Flow.
- In Targeting, open the rule that serves the Flow you want to test. The Flow's screens appear on the rule canvas, and you edit them in place. If other rules share the Flow, click the canvas background and choose Edit flow first.
- Add an experiment step in one of two ways:
- In the left nav, click Add step, then Experiment.
- Hover over the connection where the experiment should go, click +, then choose Add flow control > Experiment. The step after the connection becomes variant A (Control).
- To make the experiment the Flow's entry point, open Settings and set Initial step to the experiment.
- Set up each variant. The experiment starts with A (Control) and B. Click Add variant in Experiment settings to test more variants.
- On the canvas, click Set up variant. In Experiment settings, use the variant's Select a step dropdown instead.
- Choose Create new step, Copy from experiment variants, or Select existing step.
- To test an existing Paywall, first add it with Add step > Your paywalls.
- Set the Name and the Primary metric. The default metric is Initial conversion rate.
- Under Audience, set the Percentage of customers to enroll. The default is 100%.
- Click Start, then confirm Publish and start. Publishing the Flow starts the experiment.
Once the experiment starts, the experiment step and its variant steps are read-only. If you publish edits to a Paywall screen that's part of a running experiment, you're asked to confirm first.
Each experiment in a Flow counts as its number of variants plus 2. Multiplied across all the Flow's experiments, including ones that haven't started, the total can't exceed 64. For example, one experiment with 4 variants counts as 6, and three experiments with 3 variants each count as 125, so they can't be published together.
Pause or end an experiment
To pause, click Pause. New customers follow the holdout path, and customers already in the experiment keep their variant. Click Resume to start enrolling again.
Making a Flow inactive also pauses its running experiments. Publishing the Flow again doesn't resume them. Click Resume when you're ready.
To end the experiment, click Roll out and pick the Winning variant:
- Between Flows: choose Roll out winning flow to all customers. The rule serves the winning Flow directly, keeps its Checkpoints, audience, and schedule, and the experiment is removed.
- Inside a Flow: choose Roll out winning variant to all customers to route every customer through the winning branch, or Mark winner and update funnel manually to edit the Flow yourself. To end without a winner, pick No winner to remove the experiment.
How assignment works
- A customer is assigned a variant the first time they reach the experiment.
- The assignment is sticky. RevenueCat assigns the variant from the customer's App User ID, so they see the same variant across sessions and at every Checkpoint that serves the experiment.
- Enrolled customers are split evenly between variants until the experiment ends.
- Customers who aren't enrolled in the experiment are considered part of the holdout group, and are served the control variant. The ability to configure the experience the holdout group receives uniquely is coming soon.
View results
In Experiment settings, click See results to open the experiment's results page, which has the Results summary, Full report, and Variants setup tabs. See Experiments results for more info, including how sandbox purchases are handled.
For experiments between Flows, results include every purchase an enrolled customer makes while the experiment runs, not only purchases made in the variant Flow. Step-by-step metrics are split per variant Flow, and each variant links to its Flow or Offering.