Measure what the assistant adds with a holdout test

A holdout test answers one question: do sessions with the assistant add to the cart more often than sessions without it? While a test runs, half of the site's sessions are served no assistant. AISA compares the share of sessions with an add-to-cart in the two groups.

The test lives on the Holdout tab of the Experiments page in the sidebar, at /dashboard/sites/{siteId}/experiments/ab-test?tab=increment. The old address, /dashboard/sites/{siteId}/settings/holdout, redirects there. Every member can read it. Starting, extending and stopping need the Edit sites permission (sites:edit). Without it the tab shows the running test and the past tests, no New test button, and a note on the running test's card in place of the buttons.

What a test doesCopy Anchor Link

• The split is fixed at 50/50. There is no share to choose. A smaller control group would only make the result slower to read.
• Groups are per browser session, not per shopper. A session lands in a group when the page loads, from its session id and the test. It keeps that group for the whole test. A shopper who comes back in a new session can land in the other one.
• A session without the assistant sees nothing of it. The tag gets no engagement widget and no resume button, and the panel does not open. A visitor the site does not accept because the organization reached its conversation quota is in neither group.
• A shopper who objected to measurement is never assigned. That shopper is offered the assistant as usual and counts in neither group.
• A test has a planned duration and ends by itself. You pick 3, 7, 14 or 28 days when you start. See Start a test.
• One test runs per site at a time, and never together with a widget experiment. A test cannot start while an engagement widget experiment is running on the site, and an experiment cannot start while a test is running. Only a running experiment counts: a paused or stopped one does not block a test. The two would compete for the same sessions, since a test serves half of them no assistant. The check runs at the start only, so two members who start one of each at the very same moment could end up with both running. Both stay valid, only slower to read.

An add-to-cart counts for a session when it happens at or after the session joined its group, from either place: the assistant's panel, or your own pages through the Shopify pixel. A session counts in its group whatever it did afterwards.

A tab opened from the panel keeps the sessionCopy Anchor Link

A product link in the assistant's panel opens in a new tab, and a new tab used to start a new session, with a new draw. An add-to-cart made there could then count in the other group. Now, when a shopper opens such a link, the tag leaves the session id in the browser's localStorage on your origin as iadvize:handoff, with the time. A tab of your site that has no session of its own and loads within 15 seconds continues that session, so its add-to-cart counts in the same group. The value is not removed when read, so several links opened in a row each continue the session. After 15 seconds it is ignored, and removed the next time a tab that starts a session finds it. The panel tells the tag with a message that carries nothing, not even the link. Nothing is written for a shopper who objected to measurement. See What the tag stores in the browser.

Before you startCopy Anchor Link

• The Shopify pixel must be recording. The group without the assistant can only add to the cart on your own pages, so only the pixel can count it. A test starts only when Count add-to-carts on your pages is on in Site access and the pixel has recorded an add-to-cart on the site in the last 7 days. Otherwise the start is refused and the dialog says which of the two is missing. See Count add-to-carts and orders on your Shopify store.
• Your storefront must not open the assistant from its own code. A session without the assistant opens nothing. A button wired with data-iadvize-open, or a script calling iadvize('open'), would do nothing for half of your shoppers. See Open the assistant. AISA cannot check this for you, so the start dialog asks you to confirm it.
• The Edit sites permission. Hiding the assistant from real shoppers is a change to the site, so it needs the same permission as the other site settings.

Start a testCopy Anchor Link

1. In the sidebar, open Experiments, then the Holdout tab.
2. Click New test, at the top right. The button shows only while no test runs on the site. The dialog states what the test does and the 5,000 sessions each group needs before a verdict.
3. Pick a planned duration: 3, 7, 14 or 28 days. Each option shows the sessions each group should get and the smallest gap the test could confirm, in points and as a share of your add-to-cart rate. One option is marked Recommended and preselected.
4. Tick the confirmation that your storefront has no button or script that opens the assistant itself.
5. Click Start test.

Start test stays disabled until the box is ticked, and while a block shows in the dialog: the pixel check fails, or a widget experiment runs on the site. The reason shows in the dialog. A start without the confirmation is also refused on the server. A test that already runs on the site refuses a second start.

How the durations are sizedCopy Anchor Link

The figures come from the site's distinct sessions with an exposure over the last 7 days, and the share of them with an add-to-cart, your baseline rate. For each duration AISA takes the sessions each group would get at the smaller group's share, and computes the smallest gap the test could confirm at that size, with both groups at the baseline rate. It is an estimate: traffic that changes during the test changes the figures.

• The target. An option reaches the precision target when the smallest gap it could confirm is at most 10% of your baseline rate. For a 7.8% baseline that is about 0.78 points. This is a product choice, not a measured fact.
• The recommendation. The shortest option that reaches the target. When none does, the dialog says so and the longest option is recommended, with the gap it confirms.
• 3 days. It covers three days of the week only, so it is offered only when it reaches the target. 7 days and up cover whole weeks.
• No recent traffic or no add-to-cart observed. Nothing can be estimated. The dialog says so and preselects 14 days.

From the next page load, half of the sessions are served no assistant. The start is recorded in Admin activity as Holdout test started.

Extend a testCopy Anchor Link

While a test runs, click Extend test on its card and pick 7, 14 or 28 days. The planned end moves later by that much. The test keeps its identity, so sessions already assigned keep their group, and the group without the assistant stays without it until the new end. There is no cap on the total. An extension is refused once the test has ended, has reached its planned end, or has no planned end (see below). The extension is recorded in Admin activity.

Stop a testCopy Anchor Link

You can stop a test at any time: click Stop test on the running test's card and confirm. Every session is offered the assistant again from its next page load. The card goes away and the test moves to Past tests, where its figures stay. New test shows again at the top right. The stop is recorded as Holdout test stopped.

A test can also end without you: at its planned end, or when the split drifts (see Invalidated).

The planned endCopy Anchor Link

At its planned end the test ends by itself, with the ended reason planned end. From that instant every session is offered the assistant again, and the figures stay on the page. Nothing depends on a scheduled job: the serving path stops assigning sessions at the planned end, and the next configuration request the tag makes (or the next start of a test) marks the test ended. A read of the tab already shows the test as ended at that point, without writing anything. The end is recorded in Admin activity as Holdout test stopped, attributed to System, with the reason planned_end.

A test started before planned ends existed has no planned end. It runs until a member stops it, shows no forecast and cannot be extended.

Read the resultCopy Anchor Link

The tab has two sections. Running tests holds a compact card for the test that runs now, and shows only while one runs. Past tests lists the tests that are over. The card gives the quick read. A run view gives the detail. A site that has never run a test shows an empty state, No holdout test yet, with a hint to use New test for members who can edit sites.

The cardCopy Anchor Link

The card shows the test's status, Running, and Day N of M while a test with a planned end runs. It shows the start date, then one card per group, With the assistant and Without the assistant: the sessions assigned, how many added to the cart, and the share. Until both groups have a session the share reads "No session yet", never 0%. A progress bar shows the smaller group against the 5,000 floor, and a Verdict box gives the verdict line. View details opens the run view.

An invalidated test shows only a notice that the figures are not a result.

The filters of the report pages don't apply to these figures.

The health lineCopy Anchor Link

Under the verdict, one line states four checks. Each has an icon and a sentence, so none relies on colour alone.

• The split. Groups evenly split, or Groups unevenly split when the sizes of the two groups differ more than chance explains.
• The Shopify pixel. Shopify pixel recording, or Shopify pixel quiet for 24 hours when it recorded no storefront add-to-cart for that long.
• The widgets. Widgets shown as expected, or the name of the widget that most often did not show, with the share of its eligible sessions, when that share is above 5%.
• The assistant. Assistant unchanged since the start, or the number of audited changes made to the assistant since the start. Only changes made through the dashboard, the REST API or the MCP server are audited, so this count is a floor. It never proves that nothing else changed.

The forecast, the value added to carts and the estimate for engaged sessions are not on the card. They are in the run view.

The list of past testsCopy Anchor Link

Under Past tests, a table lists the tests of the site that are over, newest first. The running test is not in it: it has its own card. Each row shows:

• Started on: the start date, the planned duration and the assistant version live when it started. A test started before the version was recorded shows no version.
• Status: Ended.
• Sessions without / with the assistant.
• Add-to-carts without / with, and the share of sessions each is.
• Effect, in points: the share with the assistant minus the share without.
• Result, one of Collecting data, Assistant group higher, Group without the assistant higher, Not conclusive, No verdict or Invalidated.
• Validity: Valid, Uneven split, Split drifted, Assistant changed, or a mix of the last with a split state.

An invalidated test shows No figures. The figures of an ended test are the ones it froze when it ended. The table pages with its own previous and next buttons and shows 50 tests at most per page.

Two filters sit above it: Result and Started between (the period the test started in). There is no status filter, since running and past tests are in separate sections. The filters are written to the page address, so a filtered view can be shared. Choose All results or Clear period to take one off. When nothing matches, the table says No test matches these filters.

A click on a row opens the run view. The Actions menu at the end of the row has:

• View details, for every member.
• Start a new test, for members who can edit sites. It is disabled while a test runs on the site.

Extend test and Stop test are on the running test's card, not in the table.

The run viewCopy Anchor Link

A run has its own page, at /dashboard/sites/{siteId}/experiments/ab-test/increment/{holdoutId}, under the Experiments page, opened from the card, a row or the Actions menu. The old address, /dashboard/sites/{siteId}/settings/holdout/{holdoutId}, redirects there. Every member can read it. Back to Holdout tests returns to the Holdout tab. An id that does not exist, or that belongs to another site, shows a not-found page. The same members who can act on the tab get the same buttons at the top: Extend test and Stop test on a running test, Start a new test on an ended one (disabled while another test runs). Others see a note.

The page has these parts, top to bottom:

• Header. Start and end dates, the planned end or the days so far, the planned duration, why it ended (Stopped by a member, Ended at its planned end or Ended by the split check), the assistant version and the share of sessions without the assistant.
• Summary. Short fixed sentences: the two shares and the gap, whether the read is conclusive, or what the test can confirm at its size. A device that goes the same way in this test and in the site's other tests, and a test stopped early, get their own line.
• Groups and result. Sessions and add-to-cart share per group, the events per session with a cart, the effect with the range it likely lies in once both groups reach the floor, the share of sessions with the assistant that started a conversation, the progress to the floor, the smallest gap the test can confirm and, while it runs, the forecast at the planned end. A figure that does not exist yet reads Not yet.
• What was added to carts. See Value added.
• Charts. The share of sessions with an add-to-cart in each group every 6 hours, and the sessions assigned to each group per hour. The second shows when a gap between the groups built up.
• By device. The effect per device, with how many of the site's other tests went the same way. The page states that the test is sized for the whole site, so a device row is a lead and not a verdict. A device with too few sessions is marked as such.
• What the assistant group did. A funnel from sessions to add-to-carts after a conversation, per-widget and per-starter-question tables, and conversations by type and resolution. These describe the group with the assistant. They do not compare the two groups. Conversations not labeled yet are counted apart.
• Can this run be trusted? The checks behind the health line, with their detail: the split and its p-value, the pixel and the date of its last storefront add-to-cart, the widgets that did not show, and whether the test stopped before its planned end.
• Changes during the run. The audited changes made to the assistant while the test ran, with a link to Admin activity for the period. A change is reported, never prevented.

An invalidated test shows only its header, a notice, the validity checks and the changes.

Value addedCopy Anchor Link

In the run view, each group shows the add-to-cart events, the value added to carts per currency, the value per assigned session, and the median and mean per session with a cart. These figures are descriptive. They are what was added to carts, including what was later removed: not revenue, and no verdict is drawn from them. A site with several currencies gets one line per currency, never a sum across them. Amounts are recorded only from late October 2026, so older events show as without an amount and the value shown is partial. A test that began before then is not read as a test with no value.

An estimate for sessions that engageCopy Anchor Link

When the result is conclusive, the run view also shows an estimated effect among the sessions that engage with the assistant: the effect divided by the share of assigned sessions with the assistant that started a conversation. It is an estimate for those sessions, not for all sessions. It assumes a session without the assistant has no way to start a conversation. It never feeds the verdict.

ForecastCopy Anchor Link

While the test runs, the run view shows the planned end and a forecast: the smallest gap the test should confirm by that date, at the pace so far, as an estimate recomputed at every read. It shows only for a test that has a planned end, has sessions in both groups and has run for at least 24 hours, so that the pace of a few hours is not carried to the end.

The states of a testCopy Anchor Link

No verdict below 5,000 sessions per groupCopy Anchor Link

No verdict shows until both groups have 5,000 assigned sessions. Before that, the tab shows figures and progress only. The floor is the same one the engagement widget experiments use. It is stored on the test when it starts.

Inconclusive is not no differenceCopy Anchor Link

Past the floor, the test either reaches a verdict or says Not conclusive yet. The second message gives a detection limit: at this size the test can confirm a gap of N points or more. A smaller gap could be real and still not be seen. An inconclusive result never means the assistant makes no difference, and the tab never says so. Extend the test to shrink the limit.

A test that reached a verdict keeps running until its planned end or until you stop it. A conclusive result before the planned end is announced on the card, and it never stops the test. The card says an effect read this early may overstate the true effect. The verdict itself stays valid. Stopping at the first conclusive read tends to inflate the size of the effect, so let the test reach its planned end when you can.

PausedCopy Anchor Link

If the pixel records no add-to-cart on your pages for more than 24 hours while a test runs, the verdict is held and the tab says Verdict paused. A silent pixel would make the group without the assistant read zero, and the assistant would win for the wrong reason. The figures stay visible. The verdict returns when the pixel records again. Check the pixel in Shopify.

InvalidatedCopy Anchor Link

AISA checks that the two groups stay near their expected size. If the split drifts, the test ends by itself and the tab says Test ended, figures not shown. The figures of such a test are not a result, so none are shown. The check runs whenever the result is read, and on a small sample of the tag's page loads while the test runs. Start a new test. An automatic end is recorded in Admin activity as Holdout test stopped, attributed to System.

API and MCPCopy Anchor Link

The same test is available outside the dashboard, with the same permission and the same refusals. Only the confirmation box is the dashboard's: over the API and MCP, check with the merchant yourself that the storefront does not open the assistant from its own code.

• GET /api/v1/sites/{siteId}/holdouts lists the site's tests, newest first, like the table. It takes status (running or ended), outcome (collecting, treatment_higher, control_higher, not_conclusive, no_verdict or invalidated), from and to (ISO 8601, on the start date), cursor and limit (up to 50). meta.matching counts the matching tests, up to 10,000, on the first page only.
• GET /api/v1/sites/{siteId}/holdouts/{holdoutId} returns one test with what the run view shows, and 404 for a test that does not exist or belongs to another site. analysis is null for an invalidated test.
• GET /api/v1/sites/{siteId}/holdout, GET …/holdout/start-options, POST …/holdout/start, POST …/holdout/extend and POST …/holdout/stop. The start options route returns what the dialog shows: the durations with their sessions per group and confirmable gap, the recommendation, and why a start would be refused.
• The MCP tools get_site_holdout, start_site_holdout, extend_site_holdout and stop_site_holdout. get_site_holdout also returns the start options, and with a holdoutId returns that test in detail, like the run view. list_site_holdouts lists the tests with the same filters as the REST route and gives each test's holdoutId.

A start takes an optional durationDays (3, 7, 14 or 28). Without it, the start uses the recommendation for the site's traffic, and 14 days when nothing can be estimated. An extension takes days (7, 14 or 28).

A start can be refused with holdout_already_running, holdout_experiment_running or holdout_measurement_unavailable. An extension can be refused with holdout_not_extendable, and a stop with holdout_not_running. The API has no Save step: a start, an extension or a stop applies at once, like in the dashboard.