How AISA fetches your pages

AISA sends requests to your storefront to find your pages, read them and check that the tag is still installed. Every one of these requests identifies itself with the same User-Agent:

iAdvize-AISA/1.0 (+https://iadvize.ai/docs/product/site-pages/how-aisa-fetches-pages)

The link in it points to this page. The product token, iAdvize-AISA/1.0, does not change without this page changing in the same release, so a firewall rule that matches it keeps working.

What AISA requests, and whenCopy Anchor Link

An import runs when a member starts one from Pages, the REST API or the MCP server, and once during onboarding for a site that never had one. Nothing in AISA watches your pages for changes. The one exception is the hourly homepage check, which only looks for the tag. The one thing that reads a page more than once is automatic reading, which is on by default: it reads a new page once, and reads a page again after 7 days when it reaches your views threshold, or after 30 days otherwise. A request your site answers with a 503 or a 429 and a Retry-After header is repeated once, after the wait the header asks for (10 to 60 seconds).

Page-view counting makes no request to your site. The tag already on your pages reports the page views, and a page found that way is stored without being read.

Reads are bounded per site: 25,000 page reads a day (UTC), whatever started them. See Limits.

How the requests behaveCopy Anchor Link

• GET requests over https, with no cookie and no login. A sitemap address given as http is upgraded to https.
• AISA reads the HTML your server sends. It does not run JavaScript.
• Redirects are followed one at a time, and each one must stay on your site (its URL or one of its authorized domains). At most 5 per request.
• Time limits: 10 seconds for a page, 15 seconds for a sitemap document or robots.txt.
• A page is read up to 6 MB, a sitemap document up to 50 MB once decompressed. Compressed (.gz) sitemaps are accepted.
• An address that resolves to a private or local network address is never requested.

What is not covered hereCopy Anchor Link

The assistant's own web search and web page reading tools go through a third-party search service, not through this fetcher, and do not send this User-Agent.

The assistant's site page tools make no request to your site at all. They read what AISA already stored.

When your site refuses AISA's readerCopy Anchor Link

Some sites answer every request that is not a browser with a 403, a 429 or a bot-check page. When AISA's own request is refused that way, it tries the same page once more through a partner service, Linkup, before it marks the page Failed. This is on for every site and you do not switch it on. Linkup is listed in our sub-processors.

• When it runs. After AISA's own read was refused (a 403 or 429, a bot-check page, or a page with no readable text), or after a location redirect, below. Never first. A 404, a redirect to another site, a timeout or a page over the size limit would fail the same way from anywhere, so they do not go to Linkup.
• Also after a location redirect. When a store keeps redirecting AISA's reader by the visitor's location, even after a second request for the page's own market (or at once, when the read of a page lands on the home page of a market, such as /de-de), AISA reads the page once through Linkup (standard mode only). The result is kept only when it is the requested page. See Stores that redirect by location.
• What leaves AISA. The public address of the page, and nothing else: no page text, no conversation, nothing about your account. AISA checks the address first, as it does for any read: it must be on your site, use https and not point to a private or local network address.
• How it reads. Linkup fetches the page and returns its HTML. AISA reads that HTML with the same code as a direct read, so the type, text and content come out the same. JavaScript is not run. AISA asks for the standard mode first, then once for the pro mode if standard did not give a readable page.
• If both fail, the page keeps the failure AISA's own read got.

The request to your server is made by Linkup, not by AISA's reader, so it does not use the User-Agent above and an allow rule on that User-Agent does not cover it.

The hourly homepage checkCopy Anchor Link

Every hour, AISA loads the homepage of each site that has Monitor the tag on and looks for the site's public key (pk_site_...) in the HTML. The key is in the install snippet, so it is there whether or not a consent manager holds the script back. This is how AISA knows the tag was removed, by a theme update for example. See Alerts for what happens next.

• The address. The site's URL, as set on the site. It must be https. Only that page is requested.
• The request. A GET with the User-Agent above, no cookie and no login. AISA reads the HTML your server sends, without running JavaScript, up to 6 MB. It does not use the partner service described below.
• Redirects. Followed one at a time, at most 5. Unlike a page read, a redirect may leave your site, and each address must still be public. A redirect to a /password page is treated as a password page.
• A second load. When the first load finds a readable page without the key, AISA waits 3 seconds and loads it once more. The tag counts as missing only if both loads miss it. A first load that finds the key, or cannot read the page, is final and makes no second request.
• Time limit. 10 seconds per request.

Each load ends in one of three outcomes:

A page AISA cannot verify never opens an alert and never closes one. If your firewall blocks the check, you see Cannot verify on the Installation tab, not an alert and not an email. To fix it, use the allow rule in the next section.

To stop the check, switch Monitor the tag off for the site, on the Installation tab. Nothing is requested for that site after that.

robots.txt Disallow rules are not appliedCopy Anchor Link

AISA reads robots.txt for its Sitemap: lines and nothing else. It does not apply Disallow rules, for any request on this page.

robots.txt tells crawlers that explore a site on their own which paths to skip. AISA does not explore: every page it requests is one you published, asked for, or opted in to.

• Published by you. A sitemap import reads only the sitemaps your site lists or the address you give, and stores the addresses in them without reading them.
• Asked for by you. A page is read when you add it by hand or click Read, or when someone in your organization asks through the API or the MCP server.
• Opted in by you. A page found from shopper traffic is one shoppers viewed on your own storefront. It is read only if you ask, or if you switched on automatic reading for the site.

To keep pages out, use Exclusions rather than robots.txt: a rule such as checkout/* keeps those paths out of every import and every traffic run. Removing a page found by an import or from traffic also keeps it out.

Allow AISA through a firewallCopy Anchor Link

A firewall or bot-protection service (a WAF, a CDN bot rule, a "verify you are human" check) can block AISA. A blocked page ends Failed: with No readable text on the page when a bot check answers instead of the page, or with The read failed. Read again. when the firewall answers with an error. A blocked sitemap import ends with No sitemap found or Sitemap import did not finish. Before a blocked page ends Failed, AISA tries it through a partner service, which may read it anyway. The sitemap import has no such fallback.

To let AISA through, add an allow rule that matches the User-Agent:

• Match on the product token: the User-Agent contains iAdvize-AISA/. Matching the whole string also works, but breaks if the version changes.
• Scope the rule to GET requests, and if your tool allows it, to the paths AISA needs: your pages, your homepage (for the hourly check), /robots.txt and your sitemap files.
• If your tool has a "verified bots" list, AISA is not on it: add a custom rule.

This page lists no IP address range to allow. Match on the User-Agent.

After you change the rule, open Pages, select the pages that failed and click Read again, or start a new import. For the homepage check, click Check now on the Installation tab, or wait for the next hourly run.