A/B test two versions of an agent

Each agent has a new Experiment tab. Pick another saved version and AISA serves it to half of the conversations that start on your web embed, the live version to the other half. It then compares the share of conversations with a purchase in each.

• One conversation, one version. A shopper is assigned at the first message and keeps that version. A shopper who objected to measurement is never assigned. The public link is not part of the test. Starter questions and the welcome message stay the live version's.
• Purchases within 24 hours. A conversation counts as converted when an order tied to it is placed within 24 hours of its first message. Only conversations older than 24 hours decide the result. The figures are attributed to the assistant, not incremental, and refunds and cancellations are not read.
• A result only after enough conversations. The number each version needs depends on your current rate, from 2,600 at 10% to 57,300 at 0.5%. Before that, the tab shows a live curve labelled as an indication, and never a winner. A test that cannot tell the versions apart says so and gives the smallest gap it could confirm, per 1,000 conversations. It does not say the versions are equal.
• Secondary figures. Funnel steps, orders, revenue, revenue per conversation and average order value explain the result and never pick a winner.
• Ending a test. Stop it, make the candidate live, or let it end: when another version goes live, when the optional maximum per version is reached, or when the split between the two versions turns uneven, which is paused as untrustworthy.

To start one, the agent must be the one that answers on its site, have a live version and another saved version, and the site must have a recent order and a join key. Both versions are locked once the test starts. A test started while your current rate is unknown (under 1,000 conversations or 20 purchases in 28 days) shows no result until the number of conversations needed is set, which happens once your rate is known.

Everyone who can open the agent can read the tab. Starting, stopping and making a version live need Agents: Edit, and are recorded in Admin activity. Simultaneous widget tests or a site holdout can mix their effects with this one.

API and MCP: GET and POST /api/v1/sites/{siteId}/agents/{id}/experiments, GET …/experiments/{experimentId}, POST …/stop and POST …/promote, with three new error codes: agent_experiment_already_running, agent_experiment_not_running and agent_experiment_refused. The MCP server has four new tools, get_agent_experiment, start_agent_experiment, stop_agent_experiment and promote_agent_experiment, and now has 39 tools.