Overview
Experiments are Activate's A/B testing engine. Where experiences show the same thing to every matching user, experiments deliberately split traffic across two or more variations so you can measure which one produces better outcomes. Activate handles the traffic assignment, the event tracking, and the statistical analysis. You focus on the hypothesis and the variant design.
Each experiment tracks user-level assignment, meaning once a visitor is bucketed into Variation A, they stay in Variation A on every future visit. This prevents the visitor-confusion problem where the same user sees different versions on different days and invalidates the test.
Navigate to Experiments
Open your property and go to Activate > Experiments from the left sidebar. Click Add to create a new experiment. The first thing you'll pick is Mode, either Simple for a quick two-variant test or Advanced for multi-variant or MVT-style setups. Mode is locked at creation and cannot be changed later, so choose deliberately.
Create variations
Add two or more variations including a Control. The Control should represent the existing experience with no changes, so the action list is typically empty. Each other variation carries the change you want to test, for example Variation A shows a popup, Variation B shows a banner, Variation C changes the call-to-action copy.
Keep variants meaningfully different. Testing two nearly identical buttons rarely produces a statistically significant result even with heavy traffic because the real effect size is too small to detect. Bold hypotheses with clear changes give you the cleanest reads.
Set traffic allocation
Define what percentage of matching traffic sees each variation. For a classic A/B test, use 50/50. For three-way tests, 33/33/34 is standard. You can also run uneven splits, for example 80/10/10, when you want to limit exposure to experimental variants while still generating enough signal to analyze.
Traffic allocation applies only after targeting conditions match. If your experiment targets logged-in users only, the split is across logged-in users, not all site traffic. Anonymous visitors simply don't enter the experiment at all.
Define success metrics
Pick the outcomes that matter for this test: conversions, clicks, revenue, or custom events. Activate tracks these automatically per variation so you can compare performance side-by-side. You can track multiple metrics, but pick one primary metric as the tiebreaker since secondary metrics sometimes move in opposite directions across variants.
Start the experiment
Set the experiment to Active. Matching traffic immediately starts splitting according to your allocation, and user assignments persist across sessions. A returning visitor always sees the same variation they were first assigned, which is essential for test validity.
Every time a user is bucketed, Activate fires a tp_experiment_variant event containing the experiment ID and the assigned variant. Echo picks up these events and you can route them to your ad platforms or warehouse for downstream analysis.
Analyze results
Open the experiment report to see results per variation. The report surfaces z-test, p-value, and lift calculations so you can evaluate whether the difference between variants is meaningful or just noise. Wait for the primary metric to reach statistical significance before declaring a winner. Stopping early based on a trend that reverses is the most common mistake in A/B testing.
Once you have a confident winner, move the experience logic out of the experiment and into a standalone Activate experience so every visitor sees the winning treatment. Then archive the experiment to keep your workspace clean.
Control variations are always empty so they reflect the current live experience. User assignment is persistent, meaning a returning visitor always sees the same variation, which is essential for valid test results. Each assignment fires a tp_experiment_variant event you can consume downstream, and remember that Mode (Simple vs Advanced) is chosen at creation and cannot be changed later.
Troubleshooting
Traffic split looks uneven even though allocation is 50/50
In the early days of an experiment, random assignment produces choppy splits that even out over time. If you have fewer than a few thousand users in the test, expect natural variance. If the imbalance persists past tens of thousands of users, check your targeting conditions for rules that might systematically favor one variation.
Results never reach significance
Most likely, the effect size is smaller than your test can detect with current traffic. Either run longer, increase traffic by widening targeting, or redesign variants to test more dramatic changes. Testing tiny copy tweaks against heavy traffic sites can still take months to resolve.
Same user reports different variants
User assignment depends on a persistent cookie or identifier. If visitors regularly clear cookies, use multiple browsers, or browse in private mode, they can be re-bucketed. This is expected behavior but keep it in mind when interpreting heat-map or session-recording data that suggests user-level inconsistency.