A/B test two versions of an agent

An agent A/B test answers one question: does this version of the agent turn more conversations into purchases than the one that is live? While a test runs, the shoppers who start a conversation on your site talk to one of the two versions, picked at random, half and half. AISA then compares the share of conversations with a purchase for each.

You can run an agent test from two places. They are independent: neither sends you to the other.

• The Experiments section. Under Experiments in the site's navigation, the Agent tab (/dashboard/sites/{siteId}/experiments/ab-test?tab=agent) holds the New test dialog, the running tests as cards and the table of past tests. You start a test from the dialog. An agent that has a test (running or past) has a page there, at /dashboard/sites/{siteId}/experiments/ab-test/agent/{agentId}, where you read it and end it. That page has no readiness card and no start button. An agent with no test has no page: opening its address sends you back to the Agent tab.
• The agent editor. The Experiment tab of an agent (/dashboard/sites/{siteId}/agents/{agentId}/experiment) shows the screens for that agent: readiness, how to read a test, the running test, the last result and the history. It starts a test with the same start dialog.

Both need the same permissions. Every member who can see agents (Agents: View or Agents: Edit) can read the Experiments Agent tab, the agent's test page there and the editor's Experiment tab. Starting, stopping and making a version live needs Agents: Edit, whichever place you use. Without it, New test is not shown on the Agent tab, and the buttons on an agent's test page and on the editor's Experiment tab show disabled with the reason. Each action checks the permission again on the server.

What a test doesCopy Anchor Link

• Two versions. The live version (the champion) and one other saved version of the same agent. You pick the second one when you start.
• 50/50, fixed. There is no split to choose.
• One conversation, one version. A conversation is assigned on the request that carries the shopper's first message, and keeps that version for its whole life, even after the test ends or another version goes live.
• Web embed only. The web embed takes part. The public link (hosted page) does not: its conversations always get the live version.
• A shopper who objected to measurement is never assigned. That shopper is served the live version and counts in neither group.
• Starter questions and the welcome message stay the live version's. A shopper sees them before sending a first message, so there is nothing to assign yet.
• Both versions are locked. Starting a test closes the version line of each one, the same lock as a version that has been live (see Locked versions). Changes go into a new version. The lock stays after the test ends: nothing in the product reopens a version line.
• One test per agent at a time.

Before you startCopy Anchor Link

The editor's Experiment tab shows a readiness card, and Start an A/B test is enabled when every check is met. The New test dialog of the Experiments section reads the same checks (see Start a test).

A start that fails a check, or that names a version that could not go live, is refused, whichever way it is made (dashboard, API or MCP). The card also shows what the test will read:

• Your current rate. The live version's conversion rate over the last 28 days, when it has at least 1,000 conversations and 20 purchases in that window. Only conversations at least 24 hours old, on the web embed, from a shopper who did not object, with at least one shopper message are counted. A conversation with no join key counts here and in both versions of a test, and can never be credited with a purchase. Under those minimums the rate reads as unknown.
• Traffic of the last 7 days. The conversations with a shopper message on the site, not counting the last 24 hours, whichever agent answered them.

Start a testCopy Anchor Link

There are two ways in. Both end in the same start dialog.

From the Experiments section

1. Open Experiments, then the Agent tab.
2. Click New test, at the top right.
3. Choose the agent. The start fields appear in the dialog. If the agent is not ready to test, the dialog says "This agent is not ready to test yet." and lists the checks below it, each marked met or not. There is no link to another page.
4. Under Version B, choose the version to compare with the live one (Version A, live).
5. Optionally, set Stop after (see The optional maximum).
6. Check the numbers the dialog shows, then click Start the test.

From the agent editor

1. Open the agent, then the Experiment tab.
2. Click Start an A/B test, then follow the steps above from Version B.

The dialog shows What this test measures, closed by default. Click it to read the limits of a test.

The dialog gives the split, the number of conversations needed per version, and an estimated duration. The estimate is twice that number divided by the daily average of the last 7 days. It is never a reason to refuse a start: when it is unknown, or longer than 28 days, the dialog warns that the test may take a long time and lets you go on. The start is recorded in Admin activity as Agent experiment started.

The Agent tabCopy Anchor Link

The Agent tab of the Experiments section lists the site's agent tests in two sections. The editor's Experiment tab has no such list: it shows one agent.

• Running tests: one card per test running now, with its start date and a View details link to the agent's test page. The section is absent when no test runs.
• Past tests: a table of the finished tests. Filter it by Result and by Started between. A row opens the agent's test page.

A site with no agent test shows No agent test yet.

What the test measuresCopy Anchor Link

The measure is the conversion rate per assigned conversation: the share of the version's conversations credited with a purchase.

• A purchase counts within 24 hours of the first message. A conversation is credited when an order carrying its join key is placed from its first message up to 24 hours after. It counts once, however many orders it has. Test orders are left out.
• Only mature conversations decide. A conversation is mature once 24 hours have passed since it started, so its credit is final. The result, the floor and the range count mature conversations only. Other figures include the ones still maturing, and the tab says how many.
• Attributed, not incremental. The rate is what the chat is credited with. It is not extra revenue, and the test does not compare against shoppers who had no assistant. For that, see the holdout test.
• Refunds and cancellations are not read. An order that was later refunded or cancelled still counts as a purchase.

How many conversations a result needsCopy Anchor Link

The number of conversations each version needs before the test can name a winner is the floor. It depends on your current rate: the lower the rate, the more conversations it takes to see a difference. The table gives the floor for a 30% relative difference between the versions (from 2.0% to 2.6%, for example), with a 10% risk of naming a winner by chance. AISA sets it when the test starts and it never changes after.

A rate between two rows takes the row of the lower rate, so the larger floor. Below 0.5% or above 10%, the floor is an estimate worked out from the rate (about 288 × (1 − rate) ÷ rate, rounded up to the next hundred), and the dialog says so.

⚠ These floors come from a simulation, not from measured iAdvize pilot data. At low rates the test takes weeks.

Read the tabCopy Anchor Link

A live indication, not a resultCopy Anchor Link

Conversion rate over time draws each version's rate by day, with a shaded range around it. It carries the badge Indication, not a result. The range appears once each version has 200 conversations. Until then the tab says it waits for them.

Before a result, the tab states what it sees: each version's rate, the gap so far, and whether the ranges overlap. It names no winner and gives no probability. It also says the smallest gap the test could confirm with the traffic so far, so you know what "too early to tell" means in numbers.

A resultCopy Anchor Link

A result appears when both versions have reached the floor in mature conversations and the evidence is strong enough. The test keeps looking as conversations arrive, so reading it early does not weaken it. Three states:

Gaps are worded per 1,000 conversations: "about 6 more purchases for every 1,000 conversations" is the candidate's rate minus the live rate, times 1,000, rounded.

A conclusive test keeps running until you act. Nothing ends it for you.

Secondary driversCopy Anchor Link

The conversion rate is the main driver, and the only figure that picks a winner. The others explain it:

• Funnel: conversations with a message, those that saw a product, clicked a product, added to the cart through the assistant, and bought. Adds to cart made on the storefront are not counted. Seeing the cart card or tapping its Go to cart does not count as seeing or clicking a product.
• Orders and revenue: the number of orders, the revenue attributed to the assistant, revenue per conversation and average order value. Amounts are read in the currency with the most orders, and the tab says when orders came in several.

A version can win on conversion and earn less per conversation. When it does, a second banner says so. Compare revenue per conversation, not total revenue: the two versions did not get exactly the same number of conversations.

End a testCopy Anchor Link

Making a version live is always your call: the tab only offers it after a winner, but it changes what every shopper is served whatever the result says.

Once a test ends, its conversations keep the version they were assigned. That holds after every end, an uneven-split pause included: a conversation already assigned to the candidate keeps being served the candidate, while new conversations get the live version. The tab shows the last test in full, and Past tests lists up to ten earlier ones with their versions, period and outcome.

The optional maximumCopy Anchor Link

Stop after sets a number of conversations per version, at least 1,000 and at most 2,147,483,647. The test ends when both versions have that many mature conversations, whatever the result. The tab then shows the gap such a test could confirm, and says a smaller gap, if any, could not be seen. It never reads that as "the versions are equal".

⚠ The maximum, the even-split check and the setting of a missing number needed run on about one first message in 20, picked by a hash of the conversation. A test can therefore run a little past its maximum, and it only ends when new conversations arrive on the site.

LimitsCopy Anchor Link

• Simultaneous tests may mix. Widget A/B tests and the site holdout test running at the same time can mix their effects with this one. Nothing blocks you from running them together.
• Web embed conversations with a shopper message only.
• Attributed to the chat, not incremental.
• Starter questions and the welcome message are the live version's. A version that changes them is only tested on what comes after the first message.
• One test per agent. Two agents on one site cannot both be tested: only the one that answers can start a test.
• Versions are locked from the start (see above).
• Shopping assistant agents only.

API and MCPCopy Anchor Link

A test can be started, read, stopped and promoted through the REST API and the MCP server. See Agent experiments.

The API and MCP answer a test paused for an uneven split differently from the dashboard, which hides its figures. They return the state ended with the status auto_paused, the conversation counts and the secondary drivers, and no winner, no gap, no confirmable gap, no conversion rate and no range (those fields are null). The list of an agent's tests (GET .../experiments) returns a summary of each one, newest first and at most 50, with no figure: read one test for its figures. Stopping or promoting a test through the API or MCP names that test, and is refused with agent_experiment_not_running when it is no longer the one running on the agent.