Get now

robots.txt agent-user policy

User-triggered agents (ChatGPT-User, Claude-User, Perplexity-User) fetch a page because a person asked for it, so blocking them is usually an accident.

Standard
Established
77%Top 100 adoption77 of 100 sites implement it8/9OverviewStalledNo meaningful support, or vendors have said no.

Who supports it

  • ChatGPTPartial support

    Undated · last checked 9 Oct 2026: OpenAI: Overview of OpenAI CrawlersThe ChatGPT-User token is documented, but 'Because these actions are initiated by a user, robots.txt rules may not apply.'

  • ClaudeSupported (announced)

    Published 7 Apr 2026 · last checked 9 Oct 2026: Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?Claude-User is covered by Anthropic's statement that its bots honour robots.txt.

  • GeminiNot supported

    Published 19 Aug 2026 · last checked 9 Oct 2026: Google: Google's user-triggered fetchersGoogle-Agent and Google-GeminiNotebook (along with the other user-triggered fetchers) 'generally ignore robots.txt rules'.

  • PerplexityNot supported

    Published 29 Jan 2026 · last checked 9 Oct 2026: Perplexity CrawlersPerplexity-User: 'Since a user requested the fetch, this fetcher generally ignores robots.txt rules.'

  • Meta AINot supported

    Undated · last checked 9 Oct 2026: Meta: Meta Web Crawlersmeta-externalfetcher, which fetches links at a user's request, 'may bypass robots.txt rules'.

  • Claude CodePartial support

    Published 9 Oct 2026 · last checked 9 Oct 2026: Claude Code: Tools reference (WebFetch tool behavior)The docs say WebFetch sends a User-Agent beginning with 'Claude-User', so Claude-User groups match it. The Claude Code docs do not say outright that WebFetch checks robots.txt; that claim rests on Anthropic's general bot policy. The date comes from the page's dateModified metadata.

  • AmazonbotSupported (announced)

    Undated · last checked 9 Oct 2026: Amazon: AmazonbotAmzn-User (user-initiated, e.g. Alexa queries) is listed among the agents that 'respect the Robots Exclusion Protocol'.

20 others: no evidence yet. See every agent and browser

What it checks

Some agents only visit a page when a person asks them to: someone pastes a link into ChatGPT, Claude, or Perplexity and asks about it. The extension works out which robots.txt group applies to each of ChatGPT-User, Claude-User, and Perplexity-User, and whether that group blocks the whole site.

It also tells a deliberate block apart from collateral damage. A blanket User-agent: * / Disallow: / catches these agents too, and most sites that do it did not mean to.

A site with no robots.txt blocks nobody, so it passes here. The missing file is reported by robots.txt present & valid instead, so it is not counted twice.

Results

Status When
Pass None of the three agents is blocked, or there is no robots.txt
Fail One or more of them is blocked, either by name or by a blanket wildcard rule

How to fix

Allow the user-triggered agents explicitly. Blocking them takes your site out of answers people ask for, and does nothing to keep it out of training data, which uses different crawlers.

User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /

If your * group disallows everything, a named group like this one overrides it for these agents.

To control training use, write rules for the training crawlers instead. See AI bot rules in robots.txt.

Support by agent and browser

Announced means the vendor’s own docs or announcement say so. Published is the date on the vendor’s own page, where it has one. Checked is the last time our weekly research found that page still saying so.

AgentStatusSource
AI assistants
ChatGPT OpenAIPartial support

partial OpenAI: Overview of OpenAI Crawlers Undated · Checked The ChatGPT-User token is documented, but 'Because these actions are initiated by a user, robots.txt rules may not apply.'

Claude AnthropicSupported (announced)

Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? Published · Checked Claude-User is covered by Anthropic's statement that its bots honour robots.txt.

Gemini GoogleNot supported

no Google: Google's user-triggered fetchers Published · Checked Google-Agent and Google-GeminiNotebook (along with the other user-triggered fetchers) 'generally ignore robots.txt rules'.

Perplexity PerplexityNot supported

no Perplexity Crawlers Published · Checked Perplexity-User: 'Since a user requested the fetch, this fetcher generally ignores robots.txt rules.'

Meta AI MetaNot supported

no Meta: Meta Web Crawlers Undated · Checked meta-externalfetcher, which fetches links at a user's request, 'may bypass robots.txt rules'.

Coding agents
Claude Code AnthropicPartial support

partial Claude Code: Tools reference (WebFetch tool behavior) Published · Checked The docs say WebFetch sends a User-Agent beginning with 'Claude-User', so Claude-User groups match it. The Claude Code docs do not say outright that WebFetch checks robots.txt; that claim rests on Anthropic's general bot policy. The date comes from the page's dateModified metadata.

AI crawlers
Amazonbot AmazonSupported (announced)

Amazon: Amazonbot Undated · Checked Amzn-User (user-initiated, e.g. Alexa queries) is listed among the agents that 'respect the Robots Exclusion Protocol'.

No evidence yet20
  • Copilot
  • ChatGPT Atlas
  • Comet
  • Dia
  • Gemini in Chrome
  • Edge Copilot Mode
  • Cursor
  • Codex
  • GitHub Copilot
  • Windsurf
  • CCBot
  • Applebot
  • Bytespider
  • Cohere
  • Mistral
  • DuckAssist
  • Chrome
  • Edge
  • Firefox
  • Safari

Names and marks belong to their owners and are used only to identify products. None of these vendors is affiliated with Sitebulb. Spotted something out of date? Send a correction.

Adoption on the top 100 sites

77 of 100 measured sites implement it

Scanned 23 Oct 2026 with the Sitebulb Labs extension, against the Tranco list (tranco-56Z8N).

9 Oct 2026: 80%12 Oct 2026: 78%23 Oct 2026: 77%
9 Oct 2026 80% → 23 Oct 2026 77%
Sites implementing it77
  1. #1 google.com
  2. #2 cloudflare.com
  3. #3 aws.amazon.com (via amazonaws.com)
  4. #8 googleapis.com
  5. #9 microsoft.com
  6. #11 youtube.com
  7. #12 apple.com
  8. #20 dzen.ru
  9. #23 googletagmanager.com
  10. #24 azure.microsoft.com (via azure.com)
  11. #26 m365.cloud.microsoft (via office.com)
  12. #27 outlook.live.com (via live.com)
  13. #32 github.com
  14. #33 wikipedia.org
  15. #37 fastly.com (via fastly.net)
  16. #39 whatsapp.com (via whatsapp.net)
  17. #42 bing.com
  18. #44 triplinkintl.com
  19. #45 googleusercontent.com
  20. #48 gandi.net
  21. #53 icloud.com
  22. #55 wordpress.org
  23. #56 teams.live.com (via skype.com)
  24. #57 cloudflare.net
  25. #59 goo.gl
  26. #60 digicert.com
  27. #61 adobe.com
  28. #67 domains.google (via googledomains.com)
  29. #68 zoom.com (via zoom.us)
  30. #75 roblox.com
  31. #76 opera.com
  32. #77 open.spotify.com (via spotify.com)
  33. #81 tiktokv.com
  34. #83 ya.ru (via yandex.net)
  35. #84 nginx.org
  36. #85 ntp.org
  37. #87 qq.com
  38. #90 vimeo.com
  39. #93 bunny.net (via b-cdn.net)
  40. #94 blogger.com (via blogspot.com)
  41. #95 mozilla.org
  42. #96 f5.com
  43. #98 telegram.org (via t.me)
  44. #99 store.epicgames.com (via epicgames.com)
  45. #101 wordpress.com
  46. #106 sentry.io
  47. #107 letsencrypt.org (via lencr.org)
  48. #109 ui.com
  49. #110 european-union.europa.eu (via europa.eu)
  50. #114 st.okcdn.ru (via okcdn.ru)
  51. #116 bitly.com (via bit.ly)
  52. #117 ipify.org
  53. #119 nic.ru
  54. #120 samsung.com
  55. #121 office365.com
  56. #122 intuit.com
  57. #123 aliyuncs.com
  58. #127 openai.com
  59. #128 one.one.one.one (via cloudflare-dns.com)
  60. #131 pages.github.com (via github.io)
  61. #132 dropbox.com
  62. #135 discord.com (via discord.gg)
  63. #136 developer.adobe.com (via adobe.io)
  64. #139 apache.org
  65. #140 android.com
  66. #141 nih.gov
  67. #144 vk.ru
  68. #147 reg.ru
  69. #150 t-online.de
  70. #151 archive.org
  71. #152 forms.gle
  72. #153 webex.com
  73. #154 snapchat.com
  74. #156 godaddy.com
  75. #158 kaspersky.co.uk (via kaspersky.com)
  76. #159 firebase.google.com (via crashlytics.com)
  77. #160 tumblr.com

Overview

8/9

Stalled

No meaningful support, or vendors have said no.

How this rating is worked out8 of 9 points
FactorPointsWhy
Vendor support2 / 32 vendors supporting, 2 partly or planned
Top-100 adoption3 / 377% of measured sites
Spec maturity3 / 3Published standard

8 or more points is Established, 5 to 7 is Gaining traction, 2 to 4 is Speculative and under 2 is Stalled. A feature is also Stalled when more vendors have said no than yes. Read the spec.

Community insights

No community insights for this feature yet.

Quotes are from public posts, checked against the original before they appear here. Suggest an insight, or ask for yours to be removed.