Shopify Rollouts vs a Paid A/B Testing App

Shopify ships a native A/B test now, and the sentence doing the rounds is that testing is finally built in. That sentence is true of the platform and false of most stores on it. What a merchant on the entry plan actually gets is a staged rollout — a change that goes out on a schedule, to a share of visitors as an event or to everyone as a launch, and ends the way you chose in advance, reverted or kept — which never shows anybody a second version to compare against.

This page is the ladder underneath the headline. Rollouts open on Basic, the experiment type that makes a rollout an A/B test starts on Grow, and running that rollout inside one market is documented a step higher again. Under the ladder sits the second question: what the native report hands you when the test ends, and which of the gaps a paid testing app is really priced against.

Everything here comes from Shopify's own documentation and changelog and from the App Store listings themselves, read on September 1, 2 and 5, 2026.

What a Rollout Actually Is

Rollouts reached the admin in stages. The release post of March 31, 2026 announced them for the online store theme, and the entry of June 5, 2026 extended the same mechanism to checkout and customer accounts configurations. A changelog entry of October 2, 2026 added discounts, and Shopify's help now lists product catalogs as well. The current definition covers all four, and it is worth reading slowly, because it packs the schedule and the ending into two sentences.

Two things in that definition do the work. A rollout is scheduled, so a change does not have to go live the moment you finish building it. And a rollout has an ending you pick in advance, at which point the change either reverts or becomes permanent. Our BFCM checklist uses that ending to schedule a Black Friday sale as an event that rolls back at its end date.

Neither of those is a test yet — a rollout is a launch until you give it a control group, and the next section is where that happens.

If you would rather watch the feature before reading the mechanics, the walkthrough below builds one in the admin end to end — where rollouts live under Markets, how the traffic split is set, and what happens to the change once the end date arrives. It is one practitioner's demo from the WebSensePro channel, recorded in July 2026, and its framing is looser than this page in the one place that costs money: it presents the feature as free, which holds for a launch on Basic and higher but not for the A/B experiment, which needs Grow or above. Nothing here rests on the video — every figure, limit and plan threshold on this page comes from Shopify's own pages and from the listings themselves.

Launch vs Experiment

The difference is one field, and it is not a cosmetic one. You either leave the rollout as a launch, which is the default type, or turn it into an experiment by selecting Experiment in the rollout type menu and setting the traffic split in the Audience section, and that decides both what your visitors see and what your report will contain. It is also where the plan ladder bites, because of the three types only the experiment carries a plan floor above the entry tier.

What You Can Put in One Rollout

That is a tighter constraint than it sounds. You cannot stack three separate theme edits into one rollout and read them as a package, and the checkout side has no edit at all — the configuration is replaced whole or left alone. Shopify's Rollout resources and change types table splits the same ground into five named rows, and the third column — Supports content changes — is the one people miss.

Replacing the theme is the heaviest thing a rollout carries: a different theme entirely, run against the one you have today. Whether that move is worth making at all is a separate decision with its own timing, and our Horizon guide answers it in one section.

On the other side, replacing the checkout and accounts configuration is the change most merchants have never made deliberately. It is the configuration behind the checkout rather than the theme wrapped around the storefront, and what can live inside it is its own subject — our checkout extensibility guide covers that side.

The discount row turns a promotion into something a rollout can carry. A discount change activates or deactivates existing discounts, up to 250 of them in one rollout, so you can schedule a promotion or, in an experiment, test it against your store's usual discount behavior, which serves as the control. Its report carries the same metrics as any other rollout, with no revenue or orders attributed to the discount, and the results count every visitor who reaches the stage where the discount applies, eligible or not, in the control as well as the treatment.

The catalog row is how a rollout reaches price, a whole catalog at a time. A catalog change sets a catalog to Active or Draft, and setting it to Active publishes its prices and product availability, so in an experiment the treatment group sees that catalog's prices while the control sees the current state. A single product's own go-live date is set in its Publishing section instead — see how to schedule a product to go live.

The rollout itself edits no price: edits to a catalog's prices are saved right away and apply whenever the catalog is active, the change reaches no customer until the catalog is assigned to at least one market, and a catalog assigned directly to a company location can't be added.

A catalog experiment also splits your store from your other channels. An experiment applies its changes to your online store only, and external channels don't receive rollout information, so Shopify warns that ads, listings, product feeds and third-party apps might display a price that differs from what some visitors see. Channels such as Google and Meta can delist products or suspend an account for mismatched offers, which is why Shopify asks you to review its page on reach and external channels before you run a rollout that changes prices for products you advertise outside your online store.

Two more limits shape what a catalog experiment can show. Shopify's analytics page has no metrics table for a catalog change, and none of its experiment tables carries revenue, so the comparison is in rates. And if the same product sits in two applicable catalogs at different prices and neither catalog is more specific, for example two parent market catalogs, the lower price is displayed — so a catalog that raises a price can leave what those visitors see unchanged.

The Three Steps of the Ladder

The three thresholds are documented in two different places, which is why the ladder is usually described as having fewer steps than it has. The first two sit in adjacent sentences on the requirements page. The third sits in the release post; none of the six Rollouts help pages we read on October 5, 2026 names the Advanced or Plus plan at all.

Basic: Rollouts Without the Experiment

What Basic buys is the deployment half of the feature, and it is genuinely useful on its own. The ending is the part worth deciding on purpose: a launch, the default type, merges the change into your store permanently, so rolling it back is a choice you make by picking an event instead. A seasonal homepage that undoes itself the week after the season is an event, and it needs no upgrade.

What it does not buy is the comparison. A launch has no change-level traffic split: every visitor the traffic allocation and the markets make eligible gets the change, and none of them is held back on the current version, so nothing in the report is a difference between two things. That single distinction is why “Shopify has A/B testing now” is not a true sentence about a store on this plan.

Grow: the Step That Turns the A/B Test On

This is the cleanest evidence in the whole topic, because the two thresholds are stated one after the other on a single page: rollouts are available on the Basic plan or higher, and experiments are available to stores on the Grow plan or higher. One requirement for the feature, a second for its comparison mode.

The arithmetic is worth doing once, because the rest of this page refers back to it. Basic costs $39 a month and Grow costs $105. A store on the entry plan therefore pays $66 more a month to turn the comparison on — and that step buys an entire plan rather than a testing feature, so the experiment is one line of what the higher bill covers.

Advanced and Plus: One Market at a Time

The third step is documented somewhere different from the first two — in the release post rather than in the manual — and it is worth quoting in full, because the same sentence names both the use case and the plans it belongs to.

Two consequences follow. Testing a localized homepage for one region is a different purchase from testing a homepage: it needs Advanced or Plus rather than Grow. And once a rollout is tied to a market, deleting that market later does not cancel the rollout - Shopify returns it to Draft instead.

And the same two plans carry an adjacent Markets feature that is easy to confuse with this one — customizing themes and checkout and accounts pages per market, outside the rollout mechanism entirely. If you are not certain you even have a second market, our Markets guide defines the unit.

What a Rollout Will Not Touch

The change types are a closed list, and reading it as one saves an afternoon of looking for a setting that does not exist. A rollout carries a theme edit, a theme replacement, a replacement of the checkout and accounts configuration, a catalog set to Active or Draft, or discounts activated or deactivated. Anything a merchant typically wants to test that is not one of those five has a different owner, and the table names which.

The price row is the one most readers arrive with, and it is a larger subject than a missing feature: showing two prices to two shoppers has consequences for the checkout session itself. Our dynamic pricing guide works that through in the section devoted to it.

The form row is smaller but just as common. Splitting a popup or a signup form is a question about the form tool rather than about the theme, and our comparison of Shopify Forms against the paid form apps answers it where it belongs.

What the Rollout Report Tells You

There is no single rollout report. What appears depends on which type you ran and which resource you changed, and the gap between the widest and the narrowest experiment table is four rates against one.

Line up the five cases the table below sets out before you design the test, because the report is not something you can widen afterwards. Shopify documents a sixth case the table leaves out — a combined launch, which carries the launch metrics, revenue included, split across a storefront section and a checkout section.

One line on that page is worth carrying away with you: a launch and an experiment carrying the same set of changes can report different numbers for the same metric, because launches calculate each metric over time based on exposures while experiments calculate it through the experiment pipeline. Running one and then the other is therefore not a way to check yourself.

Why an Experiment Shows No Revenue

Line the tables up and the pattern is unmissable. The launch tables carry gross sales, average order value, orders and total sales over time. The three experiment tables — theme, checkout and accounts, and the combined case — carry none of them.

Shopify does not explain why the two sets differ, and this page will not invent a reason. What it means in practice is narrower than it first sounds: an experiment tells you which version moved a rate, and attributing money to that movement is a job for your own reporting once the rollout has ended. Our guide to Shopify analytics covers where those numbers live.

Nobody Declares a Winner for You

The analytics page hands you rates and stops there, and that absence is what the entire paid category is built on. Across the six Rollouts pages of Shopify's Help Center — the overview, rollout types, requirements and considerations, creating a rollout, managing rollouts, and analytics — read on October 5, 2026, we found no confidence level, no p-value, and no mechanism that declares a winner on your behalf.

The word winner does appear, twice, in the marketing copy of the two release posts: select the winning version in March and find the winner in June. Neither is followed by a method. What the analytics page hands you is rates over time, and deciding whether a difference between two rates is a real difference is left entirely to you.

Which Route Is Yours

The verdict table at the top answers by profile: find your row and go. This answers by combination, because five signals decide the route together — what you are testing, which plan you are on, what has to come out of the test, who should see the variant, and what the change is really for. Answer as your store is today rather than as you plan it.

What a Paid App Buys That Rollouts Doesn't

The honest framing for this section is that everything in it is a vendor's claim about a vendor's own product. None of it is independently verified, and none of it appears on the six Rollouts help pages we read. What is useful is the shape of the claims: read a whole category of listings and you can see precisely which native boundaries the paid tools are priced against.

The category is bigger than the rows below — Shopify's A/B testing and experiments collection published a count of 36 apps when we read it on September 5, 2026, and App Store search renders client-side, so no list assembled this way can claim to be exhaustive. One more worth naming is Kameleoon, whose listing says it supports multiple theme configurations and Shopify's new checkout extensibility, and which showed as Free on the listing when we read it on September 2, 2026.

What Each Listing Costs to Start

Prices in this category move faster than anything else on this page, so read the table as a dated snapshot rather than a price list. Two things matter more than the individual numbers: what each tier is metered on, and whether the money is billed through Shopify at all.

The Cheapest Route to a Real Split Test

Put the plan step and the entry tiers on one axis and the question stops being philosophical. For a store on the entry plan today, turning the native comparison on adds that same $66 a month; the cheapest paid entry tier in the table adds $15, and two more sit below the step.

The chart is not an argument for the shortest bar. The plan step buys an entire plan and a test that lives in the same admin as the change; the cheaper listings buy a different perimeter — the objects each listing's own page names, which overlap the native two rather than contain them — and, on the vendor's own word, a verdict at the end.

The bars also meter different things: the $15 tier is Visually's Up to 100 Orders tier, Shoplift's bar is the headline $99 on a card that also publishes a $74 floor scaling by site-wide monthly unique visitors, and a plan step is capped at nothing.

What Bites After You Press Start

Rollouts are unusually final for an admin feature, and the finality is spread across two help pages rather than gathered into one warning. Four of the rules worth knowing sit in the list below, and two more — how overlapping rollouts share traffic, and what your own hands do to a live test — follow underneath it.

• An applied rollout cannot be reverted. The same is true of a launch once it ends, because its changes apply permanently at the scheduled time.
• Archiving is final too. You cannot unarchive or restart an archived rollout, you cannot recover a deleted one, and archiving a rollout permanently deletes its theme changes.
• An active rollout can be neither archived nor deleted. You do not have to wait for its scheduled ending, though — Shopify lets you pause an active rollout or end it manually before the scheduled time, and a paused or ended rollout can then be archived or deleted. Ending one manually means choosing to roll the changes back or to apply them permanently.
• Unended rollouts block a normal publish. If rollouts touching a resource have not ended and you want to publish a new theme or checkout and accounts configuration outside the rollout system, you have to archive those rollouts first.

The sixth rule is about you rather than about the software. An experiment is a comparison between two fixed things, so changing one of them halfway through makes the comparison mean less than it looks like it means — and Shopify says so on the page where you would go to make the change.

Running Your First Experiment

None of this is a long build. The reason to walk it once on paper is that three of the five decide what the report is worth: the type and the change you pick decide which metrics it carries, and the traffic allocation is one of the two things that decide who is eligible to be measured. The start date matters for a different reason: a launch applies its changes permanently at the scheduled time.

Shopify's own instructions for creating a rollout walk the fields in order. What they will not tell you is which single change is worth the time the experiment will take, and that is the decision this page exists to help with.

The Bottom Line

The useful question was never whether Shopify has A/B testing — it does — but whether the version of it your plan carries answers the question you actually have. The documented change types are a theme edit, a theme replacement, a checkout and accounts configuration replacement, a catalog's status and a discount's activation, so outside them the native lane does not narrow — it simply stops, and the honest move is to price the tool that starts where it ends.

Frequently Asked Questions