iAdvizeDocs

Get a holdout test in detail

Returns one holdout test of the site with what the test view shows: its reading (`analysis.result`, with the effect and its always-valid `interval` once both groups reach the floor), the fixed-sentence `summary`, the add-to-cart events and the value added to carts per group and currency (median and mean per session with a cart, descriptive, never revenue), the effect by device (a lead, never a verdict; `sameSign` counts the site's other tests that reached the floor), the sessions per hour and the cumulative share every 6 hours, what the group with the assistant did (funnel, widgets, starter questions, conversations by type and resolution), the checks on its validity (the split with its p-value, the pixel, widgets that did not show, an early stop) and the audited changes made while it ran. Only changes made through the dashboard, the REST API or the MCP server are audited, so `changes` is a floor, never proof that nothing else changed. `analysis` is `null` for an invalidated test. `404` for a test that does not exist, belongs to another site or to another organization.

GET
/api/v1/sites/{siteId}/holdouts/{holdoutId}

Authorization

ApiKeyAuth
AuthorizationBearer <token>

A WorkOS org-scoped API key, created from the dashboard's organization settings page. Identifies the tenant every read and write is scoped to. Missing, malformed, unknown, revoked, expired, or non-organization-owned keys all return 401 (no distinction, so the response can't be used to probe which keys or org ids exist).

In: header

Path Parameters

siteId*string

The site id (prefixed site_). A site id that isn't yours is a 404, same as any other cross-tenant id.

holdoutId*string

The id of a holdout test of the site, from the list.

Response Body

application/json

application/problem+json

application/problem+json

application/problem+json

application/problem+json

curl -X GET "https://example.com/api/v1/sites/string/holdouts/string"
{  "data": {    "run": {      "id": "hold_01kvnwmagvf27rc9cze5rm9e1r",      "status": "ended",      "startedAt": "2026-10-28T19:40:37.000Z",      "endedAt": "2026-10-31T06:11:14.000Z",      "plannedEndAt": "2026-11-04T19:40:37.000Z",      "plannedDurationDays": 7,      "endedReason": "merchant_stopped",      "durationShare": 0.35,      "agentVersion": {        "id": "agtv_01kvnwmagvf27rc9cze5rm9e1r",        "number": 9      },      "outcome": "not_conclusive",      "control": {        "assignedSessions": 13952,        "convertedSessions": 1139,        "rate": 0.0816      },      "treatment": {        "assignedSessions": 13530,        "convertedSessions": 1030,        "rate": 0.0761      },      "effectPoints": -0.6,      "mobileEffectPoints": -0.5,      "validity": {        "unevenSplit": true,        "splitDrifted": false,        "changedDuringRun": true      }    },    "analysis": {      "result": {        "id": "hold_01kvnwmagvf27rc9cze5rm9e1r",        "siteId": "site_01kvnwmagvf27rc9cze5rm9e1r",        "status": "ended",        "state": "running",        "endedReason": "merchant_stopped",        "startedAt": "2026-10-28T19:40:37.000Z",        "endedAt": "2026-10-31T06:11:14.000Z",        "controlPercent": 50,        "trafficFloor": 5000,        "control": {          "assignedSessions": 13952,          "convertedSessions": 1139,          "rate": 0.0816,          "addToCartEvents": 4112,          "eventsWithoutAmount": 1,          "valueAdded": [            {              "currency": "USD",              "amountMinor": 20167900,              "addToCartsWithAmount": 4112            }          ],          "valuePerAssignedSession": [            {              "currency": "USD",              "amountMinor": 1445.5            }          ]        },        "treatment": {          "assignedSessions": 13530,          "convertedSessions": 1030,          "rate": 0.0761,          "addToCartEvents": 3693,          "eventsWithoutAmount": 2,          "valueAdded": [            {              "currency": "USD",              "amountMinor": 16874500,              "addToCartsWithAmount": 3693            }          ],          "valuePerAssignedSession": [            {              "currency": "USD",              "amountMinor": 1247.2            }          ]        },        "progress": 1,        "effectPoints": null,        "winner": null,        "detectionLimitPoints": 1.1,        "lastStorefrontAddAt": "2026-10-31T06:09:00.000Z",        "plannedEndAt": "2026-11-04T19:40:37.000Z",        "forecastGapPoints": null,        "engagementShare": 0.03,        "effectPerEngagedPoints": null,        "earlyResult": false      },      "interval": {        "effectPoints": -0.6,        "lowPoints": -1.6,        "highPoints": 0.5      },      "summary": [        {          "key": "rates",          "without": 8.2,          "with": 7.6,          "gapPoints": -0.6        },        {          "key": "notConclusive",          "limitPoints": 1.1        },        {          "key": "deviceRepeat",          "device": "mobile",          "direction": "lower",          "otherRuns": 3        },        {          "key": "stoppedEarly",          "durationShare": 0.35,          "progress": 1        },        {          "key": "changed",          "count": 3        }      ],      "cart": {        "control": {          "addToCartEvents": 4112,          "eventsWithoutAmount": 1,          "eventsPerSessionWithCart": 3.62,          "values": [            {              "currency": "USD",              "addToCartsWithAmount": 4112,              "valueMinor": 20167900,              "sessionsWithCart": 1137,              "medianMinor": 8060,              "meanMinor": 17740            }          ]        },        "treatment": {          "addToCartEvents": 3693,          "eventsWithoutAmount": 2,          "eventsPerSessionWithCart": 3.59,          "values": [            {              "currency": "USD",              "addToCartsWithAmount": 3693,              "valueMinor": 16874500,              "sessionsWithCart": 1029,              "medianMinor": 8300,              "meanMinor": 16400            }          ]        }      },      "devices": [        {          "device": "mobile",          "control": {            "assignedSessions": 11454,            "convertedSessions": 962          },          "treatment": {            "assignedSessions": 11195,            "convertedSessions": 882          },          "effectPoints": -0.5,          "thin": false,          "sameSign": {            "same": 3,            "of": 3          }        },        {          "device": "tablet",          "control": {            "assignedSessions": 221,            "convertedSessions": 20          },          "treatment": {            "assignedSessions": 228,            "convertedSessions": 19          },          "effectPoints": -0.7,          "thin": true,          "sameSign": {            "same": 1,            "of": 3          }        }      ],      "series": {        "hourly": [          {            "hour": "2026-10-28T19:00:00.000Z",            "control": 106,            "treatment": 77          },          {            "hour": "2026-10-28T20:00:00.000Z",            "control": 304,            "treatment": 273          }        ],        "cumulative": [          {            "at": "2026-10-28T18:00:00.000Z",            "control": 0.062,            "treatment": 0.061          },          {            "at": "2026-10-29T00:00:00.000Z",            "control": 0.0724,            "treatment": 0.0648          }        ]      },      "funnel": {        "sessions": 13530,        "widgetEligibleSessions": 12055,        "sawWidgetSessions": 10595,        "openedConversationSessions": 433,        "startedConversationSessions": 402,        "addedAfterConversationSessions": 71,        "addedFromPanelSessions": 16      },      "widgets": [        {          "widgetId": "ewgt_01kvnwmagvf27rc9cze5rm9e1r",          "name": "PDP Starters",          "eligibleSessions": 7551,          "shownSessions": 6529,          "notShownSessions": 3,          "openedSessions": 204,          "addedAfterOpenSessions": 48,          "notShownShare": 0.0004        }      ],      "starterQuestions": [        {          "starterQuestionId": "stq_01kvnwmagvf27rc9cze5rm9e1r",          "label": "Gift ideas under $50",          "shownSessions": 1834,          "tappedSessions": 42,          "addedAfterTapSessions": 24        }      ],      "conversations": {        "conversations": 406,        "withProductShown": 128,        "labeled": 406,        "types": {          "product_question:true": 211,          "discovery:true": 203        },        "resolutions": {          "resolution:resolved": 324,          "resolution:partly_resolved": 39        }      }    },    "validity": {      "split": {        "chiSquare": 6.48,        "pValue": 0.0109,        "uneven": true      },      "lastStorefrontAddAt": "2026-10-31T06:09:00.000Z",      "pixelRecording": true,      "stoppedEarly": {        "durationShare": 0.35,        "progress": 1      },      "widgetsNotShown": []    },    "changes": {      "items": [        {          "action": "agent.updated",          "occurredAt": "2026-10-29T16:35:00.000Z",          "targetLabel": "LaBelle Concierge"        }      ],      "limited": false    }  }}

List the holdout tests of a site

Returns every holdout test of the site, the running one included, newest first: what the runs table of the Holdout tab shows. Each test has its dates, planned duration, the agent version live when it started (`null` for a test started before it was recorded), the sessions and the add-to-cart share of each group, the effect in points, its `outcome` (`collecting`, `treatment_higher`, `control_higher`, `not_conclusive`, `no_verdict` or `invalidated`), the effect on mobile sessions and its validity (`unevenSplit`, `splitDrifted`, `changedDuringRun`). The figures of an ended test are the ones it froze when it ended. An invalidated test carries no figure. Groups are per browser session. An outcome never says there is no difference: past the floor without a verdict it is `not_conclusive`. `meta.matching` counts the matching tests up to 10,000 and is `null` after the first page; the cursor reads forward only.