Measure what the assistant adds with a holdout test

Site settings has a new Holdout tab. Start a test and half of the site's sessions are served no assistant. AISA then compares the share of sessions with an add-to-cart in the two groups, counting adds made in the panel and on your own pages.

• Fixed 50/50 split, per browser session. There is no share to pick. Groups are sessions, not shoppers, so a shopper who returns in a new session can land in the other group.
• No verdict before 5,000 sessions in each group. Until then the tab shows figures and progress.
• An inconclusive result is not "no difference". It states the smallest gap the test could confirm at its size.
• Paused when the Shopify pixel records no add-to-cart for over 24 hours, since the group without the assistant could only read zero. Ended automatically if the split between the groups drifts, with no figures shown.

Two things to know before you start. A test starts only when add-to-cart counting is on and the Shopify pixel has recorded an add-to-cart in the last 7 days. And your storefront must not open the assistant from its own code (iadvize('open') or data-iadvize-open): a session without the assistant opens nothing, so the dialog asks you to confirm it. Exposure counts in Reports drop by the same share while a test runs.

Starting and stopping needs the Edit sites permission, and both are recorded in Admin activity. Every member can read the result.

API and MCP: GET /api/v1/sites/{siteId}/holdout, POST …/holdout/start and POST …/holdout/stop, with three new error codes: holdout_already_running, holdout_not_running and holdout_measurement_unavailable. The MCP server has three new tools, get_site_holdout, start_site_holdout and stop_site_holdout, and now has 31 tools.