AI crawlers — who is reading your site

Crawlers do not run JavaScript, so the tracking script cannot see them — a site can be crawled by every AI company on earth and the script will swear nobody came. The only place that sees a crawler is the server that answers it, which is why this feature is a few lines at your edge rather than a checkbox.

What you get back: which AI companies read your pages, how often, which pages — and the exact robots.txt line for each bot if you want to change its access.

The plugin reports crawlers from PHP automatically — WordPress *is* your server, so nothing else is needed. See Install on WordPress.

Your site’s Crawlers tab has copy-paste snippets with your token already filled in, for Next.js middleware, a Cloudflare Worker, and a plain HTTP call from anything else. The shape is always the same: spot a bot-looking user agent, report it, never wait for the answer.

The pattern is deliberately broad and the naming happens on Asuito’s side, so a newly launched AI crawler shows up for every existing install without anyone redeploying.

// middleware — the Crawlers tab has your token filled in
const BOT = /bot|crawler|spider|gpt|claude|perplexity|bytespider|ccbot/i

const ua = request.headers.get('user-agent') ?? ''
if (BOT.test(ua)) {
  // Not awaited: this must never delay your page.
  fetch('https://asuito.com/api/crawler', {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({
      token: 'YOUR_SITE_TOKEN',
      hits: [{ ua, path: new URL(request.url).pathname }],
    }),
  }).catch(() => {})
}

Platforms that will not run your server code — Shopify, Wix, Squarespace, and friends — cannot report crawlers directly. If your domain sits behind Cloudflare, the Worker snippet works there instead. Otherwise this feature is unavailable on those platforms; everything else in Asuito still works.

Beside every bot in the table is its robots token — the name robots.txt actually recognises, which frequently differs from the brand name. Blocking is two lines: User-agent: with that token, then Disallow: /.

The trap worth knowing: Google-Extended controls whether Gemini trains on your site, and it never appears as a user agent — so it can never show up in the table, and blocking Googlebot does NOT stop it. To opt out of Gemini training: User-agent: Google-Extended then Disallow: /.

Privacy note: nothing here stores an IP address. A bot’s identity is in its user agent; its operator is a fact about the name, not the address.