Shopify Rollouts vs a Paid A/B Testing App

Shopify ships a native A/B test now, and the sentence doing the rounds is that testing is finally built in. That sentence is true of the platform and false of most stores on it. What a merchant on the entry plan actually gets is a staged launch — a change that goes out on a schedule, to a share of visitors, and ends the way you chose in advance, reverted or kept — which never shows anybody a second version to compare against.

This page is the ladder underneath the headline. Rollouts open on Basic, the experiment type that makes a rollout an A/B test starts on Grow, and running that rollout inside one market is documented a step higher again. Under the ladder sits the second question: what the native report hands you when the test ends, and which of the gaps a paid testing app is really priced against.

Everything here comes from Shopify's own documentation and changelog and from the App Store listings themselves, read on September 1, 2 and 5, 2026.

The Quick Verdict

Find your situation in the first column; everything after it is the evidence underneath.

What a Rollout Actually Is

Rollouts reached the admin in two moves. The release post of March 31, 2026 announced them for the online store theme, and the entry of June 5, 2026 extended the same mechanism to checkout and customer accounts configurations. Shopify's own definition covers both, and it is worth reading slowly, because it packs the schedule, the reach and the ending into one sentence.

Two things in that definition do the work. A rollout is scheduled, so a change does not have to go live the moment you finish building it. And a rollout has an ending you pick in advance, at which point the change either reverts or becomes permanent.

Neither of those is a test yet — a rollout is a launch until you give it a control group, and the next section is where that happens.

If you would rather hear the feature talked through before reading the mechanics, the episode below covers Shopify's native rollouts and A/B testing at length. Its guest is Aaron Glazer, credited in the episode description as Shopify's Director of Product, and the recording is from December 2025, before both changelog entries this page cites. Nothing on this page rests on it: every figure, limit and plan threshold here comes from Shopify's own pages and from the listings themselves.

Launch vs Experiment

The difference is one field, and it is not a cosmetic one. You either leave the rollout as a launch, which is the default type, or turn it into an experiment by setting the percentage of eligible visitors in the Change section, and that decides both what your visitors see and what your report will contain. It is also where the plan ladder bites, because of the two types only the experiment carries a plan floor above the entry tier.

Definitions and plan floors from Shopify's Rollout types and Requirements and considerations pages, read on September 2, 2026.

What You Can Put in One Rollout

That is a tighter constraint than it sounds. You cannot stack three separate theme edits into one rollout and read them as a package, and the checkout side has no edit at all — the configuration is replaced whole or left alone. Shopify's Rollout resources and change types table splits the same ground into three named rows, and the third column — Supports content changes — is the one people miss.

Resources, change types and content-change support from Types of rollouts and changes, read on September 2, 2026.

Replacing the theme is the heaviest thing a rollout carries: a different theme entirely, run against the one you have today. Whether that move is worth making at all is a separate decision with its own timing, and our Horizon guide answers it in one section.

On the other side, replacing the checkout and accounts configuration is the change most merchants have never made deliberately. It is the configuration behind the checkout rather than the theme wrapped around the storefront, and what can live inside it is its own subject — our checkout extensibility guide covers that side.

The Three Steps of the Ladder

The three thresholds are documented in two different places, which is why the ladder is usually described as having fewer steps than it has. The first two sit in adjacent sentences on the requirements page. The third sits in the release post; none of the six Rollouts help pages we read on September 2, 2026 names the Advanced or Plus plan at all.

Monthly plan prices from Shopify pricing; annual rates are deliberately not quoted here. Thresholds from the Rollouts requirements page and from the March 31, 2026 release post.

Basic: Rollouts Without the Experiment

What Basic buys is the deployment half of the feature, and it is genuinely useful on its own. The ending is the part worth deciding on purpose: Shopify sets a rollout to revert by default, so keeping the change is a choice you have to make rather than one that happens to you. A seasonal homepage that undoes itself the week after the season is a launch, and it needs no upgrade.

What it does not buy is the comparison. A launch has no change-level traffic split: every visitor the reach and the markets make eligible gets the change, and none of them is held back on the current version, so nothing in the report is a difference between two things. That single distinction is why “Shopify has A/B testing now” is not a true sentence about a store on this plan.

Grow: the Step That Turns the A/B Test On

This is the cleanest evidence in the whole topic, because the two thresholds are stated one after the other on a single page: rollouts are available on the Basic plan or higher, and experiments are available to stores on the Grow plan or higher. One requirement for the feature, a second for its comparison mode.

The arithmetic is worth doing once, because the rest of this page refers back to it. Basic costs $39 a month and Grow costs $105. A store on the entry plan therefore pays $66 more a month to turn the comparison on — and that step buys an entire plan rather than a testing feature, so the experiment is one line of what the higher bill covers.

Advanced and Plus: One Market at a Time

The third step is documented somewhere different from the first two — in the release post rather than in the manual — and it is worth quoting in full, because the same sentence names both the use case and the plans it belongs to.

Two consequences follow. Testing a localized homepage for one region is a different purchase from testing a homepage: it needs Advanced or Plus rather than Grow.

And the same two plans carry an adjacent Markets feature that is easy to confuse with this one — customizing themes and checkout and accounts pages per market, outside the rollout mechanism entirely. If you are not certain you even have a second market, our Markets guide defines the unit.

What a Rollout Will Not Touch

The change types are a closed list, and reading it as one saves an afternoon of looking for a setting that does not exist. A rollout carries a theme edit, a theme replacement, or a replacement of the checkout and accounts configuration. Anything a merchant typically wants to test that is not one of those three has a different owner, and the table names which.

Quoted restrictions from the Rollouts requirements and considerations page; change types from Types of rollouts and changes. Both read on September 2, 2026.

The price row is the one most readers arrive with, and it is a larger subject than a missing feature: showing two prices to two shoppers has consequences for the checkout session itself. Our dynamic pricing guide works that through in the section devoted to it.

The form row is smaller but just as common. Splitting a popup or a signup form is a question about the form tool rather than about the theme, and our comparison of Shopify Forms against the paid form apps answers it where it belongs.

What the Rollout Report Tells You

There is no single rollout report. What appears depends on which type you ran and which resource you changed, and the gap between the widest and the narrowest experiment table is four rates against one.

Line up the five cases the table below sets out before you design the test, because the report is not something you can widen afterwards. Shopify documents a sixth case the table leaves out — a combined launch, which carries the launch metrics, revenue included, split across a storefront section and a checkout section.

Metric tables from the Rollout analytics page in the Shopify Help Center, read on September 2, 2026.

One line on that page is worth carrying away with you: a launch and an experiment carrying the same set of changes can report different numbers for the same metric, because launches calculate each metric over time based on exposures while experiments calculate it through the experiment pipeline. Running one and then the other is therefore not a way to check yourself.

Why an Experiment Shows No Revenue

Line the tables up and the pattern is unmissable. The launch tables carry gross sales, average order value, orders and total sales over time. The three experiment tables — theme, checkout and accounts, and the combined case — carry none of them.

Shopify does not explain why the two sets differ, and this page will not invent a reason. What it means in practice is narrower than it first sounds: an experiment tells you which version moved a rate, and attributing money to that movement is a job for your own reporting once the rollout has ended. Our guide to Shopify analytics covers where those numbers live.

Nobody Declares a Winner for You

The analytics page hands you rates and stops there, and that absence is what the entire paid category is built on. Across the six Rollouts pages of Shopify's Help Center — the overview, rollout types, requirements and considerations, creating a rollout, managing rollouts, and analytics — read on September 2, 2026, we found no confidence level, no p-value, and no mechanism that declares a winner on your behalf.

The word winner does appear, twice, in the marketing copy of the two release posts: select the winning version in March and find the winner in June. Neither is followed by a method. What the analytics page hands you is rates over time, and deciding whether a difference between two rates is a real difference is left entirely to you.

Which Route Is Yours

The verdict table at the top answers by profile: find your row and go. This answers by combination, because five signals decide the route together — what you are testing, which plan you are on, what has to come out of the test, who should see the variant, and what the change is really for. Answer as your store is today rather than as you plan it.

What a Paid App Buys That Rollouts Doesn't

The honest framing for this section is that everything in it is a vendor's claim about a vendor's own product. None of it is independently verified, and none of it appears on the six Rollouts help pages we read. What is useful is the shape of the claims: read a whole category of listings and you can see precisely which native boundaries the paid tools are priced against.

The category is bigger than the rows below — Shopify's A/B testing and experiments collection published a count of 36 apps when we read it on September 5, 2026, and App Store search renders client-side, so no list assembled this way can claim to be exhaustive. One more worth naming is Kameleoon, whose listing says it supports multiple theme configurations and Shopify's new checkout extensibility, and which showed as Free on the listing when we read it on September 2, 2026.

What Each Listing Costs to Start

Prices in this category move faster than anything else on this page, so read the table as a dated snapshot rather than a price list. Two things matter more than the individual numbers: what each tier is metered on, and whether the money is billed through Shopify at all.

Tiers and feature descriptions read on each listing's Shopify App Store page on September 2, 2026 and re-checked on September 5, 2026. A snapshot of what those listings published on those days, not a ranking and not an exhaustive list of the category.

The Cheapest Route to a Real Split Test

Put the plan step and the entry tiers on one axis and the question stops being philosophical. For a store on the entry plan today, turning the native comparison on adds that same $66 a month; the cheapest paid entry tier in the table adds $15, and two more sit below the step.

The chart is not an argument for the shortest bar. The plan step buys an entire plan and a test that lives in the same admin as the change; the cheaper listings buy a different perimeter — the objects each listing's own page names, which overlap the native two rather than contain them — and, on the vendor's own word, a verdict at the end.

The bars also meter different things: the $15 tier is Visually's Up to 100 Orders tier, Shoplift's bar is the headline $99 on a card that also publishes a $74 floor scaling by site-wide monthly unique visitors, and a plan step is capped at nothing.

What Bites After You Press Start

Rollouts are unusually final for an admin feature, and the finality is spread across two help pages rather than gathered into one warning. Four of the rules worth knowing sit in the list below, and two more — how overlapping rollouts share traffic, and what your own hands do to a live test — follow underneath it.

• An applied rollout cannot be reverted. The same is true if you launch immediately, or set the rollout to apply its changes permanently when it ends.
• Archiving is final too. You cannot unarchive or restart an archived rollout, you cannot recover a deleted one, and archiving a rollout permanently deletes its theme changes.
• An active rollout can be neither archived nor deleted. You do not have to wait for its scheduled ending, though — Shopify lets you pause an active rollout or end it manually before the scheduled time, and a paused or ended rollout can then be archived or deleted. Ending one manually means choosing to roll the changes back or to apply them permanently.
• Unended rollouts block a normal publish. If rollouts touching a resource have not ended and you want to publish a new theme or checkout and accounts configuration outside the rollout system, you have to archive those rollouts first.

The sixth rule is about you rather than about the software. An experiment is a comparison between two fixed things, so changing one of them halfway through makes the comparison mean less than it looks like it means — and Shopify says so on the page where you would go to make the change.

Running Your First Experiment

None of this is a long build. The reason to walk it once on paper is that four of the five decide what the report is worth: the type and the change you pick decide which metrics it carries, the reach is one of the two things that decide who is eligible to be measured, and the timing decides whether there is a report at all.

Shopify's own instructions for creating a rollout walk the fields in order. What they will not tell you is which single change is worth the time the experiment will take, and that is the decision this page exists to help with.

The Bottom Line

The useful question was never whether Shopify has A/B testing — it does — but whether the version of it your plan carries answers the question you actually have. The documented change types are a theme edit, a theme replacement, and a checkout and accounts configuration replacement, so outside them the native lane does not narrow — it simply stops, and the honest move is to price the tool that starts where it ends.

Frequently Asked Questions