Get now

robots.txt present & valid

A reachable robots.txt lets agents discover crawl rules and sitemaps.

Standard
Established
87%Top 100 adoption87 of 100 sites implement it9/9OverviewEstablishedWidely supported and widely deployed.

Who supports it

  • ChatGPTPartial support

    Undated · last checked 9 Oct 2026: Overview of OpenAI CrawlersGPTBot and OAI-SearchBot are controlled via robots.txt, but for ChatGPT-User: 'Because these actions are initiated by a user, robots.txt rules may not apply.'

  • ClaudeSupported (announced)

    Published 7 Apr 2026 · last checked 9 Oct 2026: Does Anthropic crawl data from the web, and how can site owners block the crawler?ClaudeBot, Claude-User and Claude-SearchBot: 'Anthropic's Bots respect "do not crawl" signals by honoring industry standard directives in robots.txt.' Crawl-delay is also supported.

  • GeminiPartial support

    Published 19 Aug 2026 · last checked 9 Oct 2026: Google's user-triggered fetchersThe Google-Extended token (doc updated 2026-07-14, https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) controls use of content for Gemini training and grounding. User-triggered fetchers such as Google-Agent and Gemini Notebook 'generally ignore robots.txt rules.'

  • PerplexityPartial support

    Undated · last checked 9 Oct 2026: Perplexity CrawlersPerplexityBot is controlled via robots.txt, but Perplexity-User: 'Since a user requested the fetch, this fetcher generally ignores robots.txt rules.'

  • Meta AIPartial support

    Undated · last checked 9 Oct 2026: Meta Web Crawlersmeta-externalagent respects robots.txt. meta-externalfetcher, which fetches links on a user's request, 'may bypass robots.txt'.

  • Claude CodePartial support

    Undated · last checked 9 Oct 2026: Claude Code tools reference: WebFetchWebFetch 'sets a User-Agent header beginning with Claude-User', and Anthropic says Claude-User honours robots.txt. The Claude Code docs do not themselves say that WebFetch checks robots.txt.

  • CCBotSupported (announced)

    Undated · last checked 9 Oct 2026: Common Crawl FAQ'CCBot is an automated crawler, checking first the robots.txt...' Also obeys Crawl-delay.

  • ApplebotSupported (announced)

    Published 4 Sept 2026 · last checked 9 Oct 2026: About Applebot'Applebot respects standard robots.txt directives in general search crawls that are targeted at Applebot.' Applebot-Extended controls opt-out from foundation model training.

  • AmazonbotSupported (announced)

    Undated · last checked 9 Oct 2026: AmazonbotHonours user-agent and allow/disallow directives but not crawl-delay. Amzn-SearchBot follows rules for other search bots if not named. Amzn-User 'may not follow all robots.txt directives'.

  • CohereSupported (announced)

    Undated · last checked 9 Oct 2026: Cohere Web CrawlersPolicy commitment only: Cohere says it does not currently crawl for training and that it is 'policy to require that crawlers be designed to respect robots.txt'. Third parties report cohere-ai / cohere-training-data-crawler user agents in the wild, which these docs do not list.

  • MistralSupported (announced)

    Undated · last checked 9 Oct 2026: Mistral crawlers'Mistral uses specific robots.txt tags' (MistralAI-User, MistralAI-Index, MistralAI-Training). Only MistralAI-Training is explicitly described as disallowable.

  • DuckAssistSupported (announced)

    Undated · last checked 9 Oct 2026: DuckAssistBotA Disallow for DuckAssistBot takes effect after 72 hours.

15 others: no evidence yet. See every agent and browser

What it checks

The extension fetches /robots.txt from the site’s origin and parses it. It passes when the file is reachable and contains at least one User-agent group or Sitemap line. An agent reads this file first to learn what it may crawl and where the sitemap lives.

Results

Status When
Pass robots.txt is reachable and has at least one User-agent group or Sitemap line
Warn The file is reachable but has no User-agent groups or Sitemap lines
Fail No robots.txt was found (a 404, another error, or no response)

How to fix

Serve a plain-text file at /robots.txt on the site root. A minimal one that allows everything and points to the sitemap:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Several other checks read the same file, so getting it right helps them too: Sitemap available, AI bot rules in robots.txt, and robots.txt agent-user policy.

Support by agent and browser

Announced means the vendor’s own docs or announcement say so. Published is the date on the vendor’s own page, where it has one. Checked is the last time our weekly research found that page still saying so.

AgentStatusSource
AI assistants
ChatGPT OpenAIPartial support

partial Overview of OpenAI Crawlers Undated · Checked GPTBot and OAI-SearchBot are controlled via robots.txt, but for ChatGPT-User: 'Because these actions are initiated by a user, robots.txt rules may not apply.'

Claude AnthropicSupported (announced)

Does Anthropic crawl data from the web, and how can site owners block the crawler? Published · Checked ClaudeBot, Claude-User and Claude-SearchBot: 'Anthropic's Bots respect "do not crawl" signals by honoring industry standard directives in robots.txt.' Crawl-delay is also supported.

Gemini GooglePartial support

partial Google's user-triggered fetchers Published · Checked The Google-Extended token (doc updated 2026-07-14, https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) controls use of content for Gemini training and grounding. User-triggered fetchers such as Google-Agent and Gemini Notebook 'generally ignore robots.txt rules.'

Perplexity PerplexityPartial support

partial Perplexity Crawlers Undated · Checked PerplexityBot is controlled via robots.txt, but Perplexity-User: 'Since a user requested the fetch, this fetcher generally ignores robots.txt rules.'

Meta AI MetaPartial support

partial Meta Web Crawlers Undated · Checked meta-externalagent respects robots.txt. meta-externalfetcher, which fetches links on a user's request, 'may bypass robots.txt'.

Coding agents
Claude Code AnthropicPartial support

partial Claude Code tools reference: WebFetch Undated · Checked WebFetch 'sets a User-Agent header beginning with Claude-User', and Anthropic says Claude-User honours robots.txt. The Claude Code docs do not themselves say that WebFetch checks robots.txt.

AI crawlers
CCBot Common CrawlSupported (announced)

Common Crawl FAQ Undated · Checked 'CCBot is an automated crawler, checking first the robots.txt...' Also obeys Crawl-delay.

Applebot AppleSupported (announced)

About Applebot Published · Checked 'Applebot respects standard robots.txt directives in general search crawls that are targeted at Applebot.' Applebot-Extended controls opt-out from foundation model training.

Amazonbot AmazonSupported (announced)

Amazonbot Undated · Checked Honours user-agent and allow/disallow directives but not crawl-delay. Amzn-SearchBot follows rules for other search bots if not named. Amzn-User 'may not follow all robots.txt directives'.

Cohere CohereSupported (announced)

Cohere Web Crawlers Undated · Checked Policy commitment only: Cohere says it does not currently crawl for training and that it is 'policy to require that crawlers be designed to respect robots.txt'. Third parties report cohere-ai / cohere-training-data-crawler user agents in the wild, which these docs do not list.

Mistral Mistral AISupported (announced)

Mistral crawlers Undated · Checked 'Mistral uses specific robots.txt tags' (MistralAI-User, MistralAI-Index, MistralAI-Training). Only MistralAI-Training is explicitly described as disallowable.

DuckAssist DuckDuckGoSupported (announced)

DuckAssistBot Undated · Checked A Disallow for DuckAssistBot takes effect after 72 hours.

No evidence yet15
  • Copilot
  • ChatGPT Atlas
  • Comet
  • Dia
  • Gemini in Chrome
  • Edge Copilot Mode
  • Cursor
  • Codex
  • GitHub Copilot
  • Windsurf
  • Bytespider
  • Chrome
  • Edge
  • Firefox
  • Safari

Names and marks belong to their owners and are used only to identify products. None of these vendors is affiliated with Sitebulb. Spotted something out of date? Send a correction.

Adoption on the top 100 sites

87 of 100 measured sites implement it

Scanned 23 Oct 2026 with the Sitebulb Labs extension, against the Tranco list (tranco-56Z8N).

9 Oct 2026: 85%12 Oct 2026: 85%23 Oct 2026: 87%
9 Oct 2026 85% → 23 Oct 2026 87%
Sites implementing it87
  1. #1 google.com
  2. #2 cloudflare.com
  3. #3 aws.amazon.com (via amazonaws.com)
  4. #4 facebook.com
  5. #5 gstatic.com
  6. #9 microsoft.com
  7. #11 youtube.com
  8. #12 apple.com
  9. #13 instagram.com
  10. #15 mail.ru
  11. #16 x.com (via twitter.com)
  12. #18 linkedin.com
  13. #20 dzen.ru
  14. #24 azure.microsoft.com (via azure.com)
  15. #27 outlook.live.com (via live.com) (with warnings)
  16. #29 amazon.com
  17. #32 github.com
  18. #37 fastly.com (via fastly.net)
  19. #39 whatsapp.com (via whatsapp.net)
  20. #42 bing.com
  21. #43 netflix.com
  22. #44 triplinkintl.com
  23. #48 gandi.net
  24. #50 marketingplatform.google.com (via doubleclick.net)
  25. #52 uk.pinterest.com (via pinterest.com)
  26. #53 icloud.com
  27. #55 wordpress.org
  28. #56 teams.live.com (via skype.com)
  29. #57 cloudflare.net
  30. #60 digicert.com
  31. #61 adobe.com
  32. #63 tiktok.com
  33. #67 domains.google (via googledomains.com)
  34. #68 zoom.com (via zoom.us)
  35. #69 uk.yahoo.com (via yahoo.com)
  36. #70 sso.myfritz.net (via myfritz.net)
  37. #71 endpoints.news
  38. #72 api.whatsapp.com (via wa.me)
  39. #75 roblox.com
  40. #76 opera.com
  41. #77 open.spotify.com (via spotify.com)
  42. #80 baidu.com
  43. #83 ya.ru (via yandex.net)
  44. #84 nginx.org
  45. #85 ntp.org
  46. #86 msn.com
  47. #87 qq.com
  48. #90 vimeo.com
  49. #91 primevideo.com (via amazonvideo.com)
  50. #93 bunny.net (via b-cdn.net)
  51. #94 blogger.com (via blogspot.com)
  52. #95 mozilla.org
  53. #96 f5.com
  54. #99 store.epicgames.com (via epicgames.com)
  55. #101 wordpress.com
  56. #103 reddit.com
  57. #106 sentry.io
  58. #107 letsencrypt.org (via lencr.org)
  59. #109 ui.com
  60. #110 european-union.europa.eu (via europa.eu)
  61. #112 chatgpt.com
  62. #116 bitly.com (via bit.ly)
  63. #117 ipify.org
  64. #119 nic.ru (with warnings)
  65. #120 samsung.com
  66. #122 intuit.com
  67. #127 openai.com
  68. #128 one.one.one.one (via cloudflare-dns.com) (with warnings)
  69. #130 registrar-servers.com
  70. #131 pages.github.com (via github.io)
  71. #132 dropbox.com
  72. #133 skybridge.click
  73. #135 discord.com (via discord.gg)
  74. #136 developer.adobe.com (via adobe.io)
  75. #139 apache.org
  76. #140 android.com
  77. #141 nih.gov
  78. #144 vk.ru
  79. #147 reg.ru (with warnings)
  80. #150 t-online.de
  81. #151 archive.org
  82. #153 webex.com
  83. #154 snapchat.com
  84. #158 kaspersky.co.uk (via kaspersky.com)
  85. #159 firebase.google.com (via crashlytics.com)
  86. #160 tumblr.com
  87. #161 nytimes.com

Overview

9/9

Established

Widely supported and widely deployed.

How this rating is worked out9 of 9 points
FactorPointsWhy
Vendor support3 / 37 vendors supporting, 5 partly or planned
Top-100 adoption3 / 387% of measured sites
Spec maturity3 / 3Published standard

8 or more points is Established, 5 to 7 is Gaining traction, 2 to 4 is Speculative and under 2 is Stalled. A feature is also Stalled when more vendors have said no than yes. Read the spec.

Community insights

No community insights for this feature yet.

Quotes are from public posts, checked against the original before they appear here. Suggest an insight, or ask for yours to be removed.