Bot access control
See how the extension checks crawl rules, content signals and bot authentication to understand which AI agents can access your site.
These checks read robots.txt for what it says about AI crawlers and about
agents acting for a person, and look for the newer ways a site can state how its
content may be used or verify which bots are who they say they are.
Bot access control carries a weight of 3 in the overall score. Along with Content accessibility, it measures a deliberate decision a site has made about agents rather than something it gets for free.
Checks in this category
- AI bot rules in robots.txt
Explicit rules for AI crawlers (GPTBot, ClaudeBot, Google-Extended and others) show that someone has decided who gets in. Top 100 adoption 77% - robots.txt agent-user policy
User-triggered agents (ChatGPT-User, Claude-User, Perplexity-User) fetch a page because a person asked for it, so blocking them is usually an accident. Top 100 adoption 77% - Emerging checkContent Signals
Cloudflare Content Signals (Content-Signal in robots.txt) declare how content may be used for search, AI input, and AI training. Top 100 adoption 4% - Emerging checkWeb Bot Auth
Web Bot Auth lets verified agents prove who they are with HTTP Message Signatures, listed in a signatures directory. Top 100 adoption 13%