Undated · last checked 9 Oct 2026: OpenAI: Overview of OpenAI CrawlersTokens: GPTBot (training), OAI-SearchBot (ChatGPT search). The user-triggered ChatGPT-User fetcher is the exception: 'robots.txt rules may not apply'.
Published 14 Jul 2026 · last checked 9 Oct 2026: Google: Google's common crawlersToken: Google-Extended, a product token controlling whether crawled content is used for Gemini training and grounding (it is not a separate crawler). Google's user-triggered fetchers (Google-Agent, Google-GeminiNotebook) 'generally ignore robots.txt rules'.
PerplexitySupported (announced)
Published 29 Jan 2026 · last checked 9 Oct 2026: Perplexity CrawlersToken: PerplexityBot (search index; honours robots.txt). Perplexity-User 'generally ignores robots.txt rules' because a user requested the fetch. The date comes from the page's dateModified metadata.
Undated · last checked 9 Oct 2026: Meta: Meta Web CrawlersToken: meta-externalagent (AI training/products), with a robots.txt example; meta-webindexer is listed for Meta AI search. meta-externalfetcher (user-requested) 'may bypass robots.txt rules'.
CCBotSupported (announced)
Published 25 Sept 2026 · last checked 9 Oct 2026: Common Crawl: CCBotToken: CCBot. The date is the HTTP Last-Modified header.
ApplebotSupported (announced)
Published 4 Sept 2026 · last checked 9 Oct 2026: Apple: About ApplebotTokens: Applebot (search) and Applebot-Extended (controls use for generative AI training; it does not crawl).
AmazonbotSupported (announced)
Undated · last checked 9 Oct 2026: Amazon: AmazonbotTokens: Amazonbot, Amzn-SearchBot, Amzn-User. Amazon says they 'respect the Robots Exclusion Protocol'.
CoherePartial support
Undated · last checked 9 Oct 2026: Cohere: Cohere Web CrawlersCohere says it does not currently crawl for training and that its policy is for crawlers to respect robots.txt. Its blocking example uses the token 'Coherebot', not 'cohere-ai' (the token the extension checks).
MistralSupported (announced)
Undated · last checked 9 Oct 2026: Mistral AI: RobotsToken: MistralAI-Training ('Webmasters can disallow this user agent in their robots.txt file'). MistralAI-Index and MistralAI-User (Vibe user requests) are also documented, but the page does not say whether they honour robots.txt.
DuckAssistSupported (announced)
Published 2 Oct 2026 · last checked 9 Oct 2026: DuckDuckGo Help: DuckAssistBotToken: DuckAssistBot. A disallow takes effect within 72 hours. The date is the HTTP Last-Modified header.
The extension reads /robots.txt and looks for User-agent groups that name
known AI crawlers. It then works out whether each named crawler can reach the
site or is blocked from all of it.
The crawlers it recognises are GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI);
ClaudeBot, Claude-Web, Claude-User and anthropic-ai (Anthropic);
Google-Extended; GoogleOther; PerplexityBot and Perplexity-User; CCBot (Common
Crawl); Bytespider (ByteDance); Amazonbot; Applebot-Extended;
Meta-ExternalAgent; and cohere-ai.
This is the heaviest check in the extension. It is the only one that measures
what a site has actually decided about AI agents. Naming crawlers only to block
all of them does not earn a pass: the posture is what counts, not the naming.
A group “blocks everything” only when it disallows / without an Allow: /
to override it. An empty Disallow: line allows everything.
Results
Status
When
Pass
At least one AI crawler is named, and at least one of the named crawlers can reach the site
Warn
AI crawlers are named, but every one of them is fully blocked
Warn
No AI crawler is named and nothing blocks them (only a * group, or no groups at all)
Fail
No AI crawler is named and the * group disallows the whole site
Fail
There is no robots.txt, so no rules can be expressed
How to fix
Add a User-agent group for each AI crawler you have an opinion about, stating
what it may and may not read. For example, to allow search and answer engines
while opting out of model training:
If the whole site is blocked by User-agent: * / Disallow: /, check whether
that is intended. It is often a staging robots.txt that shipped by mistake.
If you block every named AI crawler on purpose, the warning is expected. It is
there to confirm the choice was deliberate.
Related: robots.txt agent-user policy
covers the agents that fetch a page because a person asked, and
Content Signals lets you say
how content may be used rather than only whether it may be fetched.
Support by agent and browser
Announced means the vendor’s own docs or announcement say so. Published is the date on the vendor’s own page, where it has one. Checked is the last time our weekly research found that page still saying so.
Agent
Status
Source
AI assistants
ChatGPT OpenAI
Supported (announced)
OpenAI: Overview of OpenAI CrawlersUndated · Checked Tokens: GPTBot (training), OAI-SearchBot (ChatGPT search). The user-triggered ChatGPT-User fetcher is the exception: 'robots.txt rules may not apply'.
Google: Google's common crawlersPublished · Checked Token: Google-Extended, a product token controlling whether crawled content is used for Gemini training and grounding (it is not a separate crawler). Google's user-triggered fetchers (Google-Agent, Google-GeminiNotebook) 'generally ignore robots.txt rules'.
Perplexity Perplexity
Supported (announced)
Perplexity CrawlersPublished · Checked Token: PerplexityBot (search index; honours robots.txt). Perplexity-User 'generally ignores robots.txt rules' because a user requested the fetch. The date comes from the page's dateModified metadata.
Meta: Meta Web CrawlersUndated · Checked Token: meta-externalagent (AI training/products), with a robots.txt example; meta-webindexer is listed for Meta AI search. meta-externalfetcher (user-requested) 'may bypass robots.txt rules'.
AI crawlers
CCBot Common Crawl
Supported (announced)
Common Crawl: CCBotPublished · Checked Token: CCBot. The date is the HTTP Last-Modified header.
Applebot Apple
Supported (announced)
Apple: About ApplebotPublished · Checked Tokens: Applebot (search) and Applebot-Extended (controls use for generative AI training; it does not crawl).
Amazonbot Amazon
Supported (announced)
Amazon: AmazonbotUndated · Checked Tokens: Amazonbot, Amzn-SearchBot, Amzn-User. Amazon says they 'respect the Robots Exclusion Protocol'.
Cohere Cohere
Partial support
partialCohere: Cohere Web CrawlersUndated · Checked Cohere says it does not currently crawl for training and that its policy is for crawlers to respect robots.txt. Its blocking example uses the token 'Coherebot', not 'cohere-ai' (the token the extension checks).
Mistral Mistral AI
Supported (announced)
Mistral AI: RobotsUndated · Checked Token: MistralAI-Training ('Webmasters can disallow this user agent in their robots.txt file'). MistralAI-Index and MistralAI-User (Vibe user requests) are also documented, but the page does not say whether they honour robots.txt.
DuckAssist DuckDuckGo
Supported (announced)
DuckDuckGo Help: DuckAssistBotPublished · Checked Token: DuckAssistBot. A disallow takes effect within 72 hours. The date is the HTTP Last-Modified header.
No evidence yet15
ChatGPT Atlas
Comet
Dia
Gemini in Chrome
Edge Copilot Mode
Claude Code
Cursor
Codex
GitHub Copilot
Windsurf
Bytespider
Chrome
Edge
Firefox
Safari
Names and marks belong to their owners and are used only to identify products. None of these vendors is affiliated with Sitebulb. Spotted something out of date? Send a correction.
Adoption on the top 100 sites
77%
77 of 100 measured sites implement it
Scanned 23 Oct 2026 with the Sitebulb Labs extension, against the Tranco list (tranco-56Z8N).
8 or more points is Established, 5 to 7 is Gaining traction, 2 to 4 is Speculative and under 2 is Stalled. A feature is also Stalled when more vendors have said no than yes. Read the spec.
Community insights
Sceptical
We see continued evidence that Perplexity is repeatedly modifying their user agent and changing their source ASNs to hide their crawling activity, as well as ignoring — or sometimes failing to even fetch — robots.txt files.
Gabriel CorralCloudflareOn their blog, 4 Aug 2025Website
Xe IasoCreator of Anubis, TecharoOn their blog, 14 May 2026Website
Ryo SuzukiClassmethodOn their blog, 24 Feb 2025Website
Drew DeVaultFounder, SourceHutOn their blog, 17 Mar 2025Website