Bot access control

See how the extension checks crawl rules, content signals and bot authentication to understand which AI agents can access your site.

These checks read robots.txt for what it says about AI crawlers and about agents acting for a person, and look for the newer ways a site can state how its content may be used or verify which bots are who they say they are.

Bot access control carries a weight of 3 in the overall score. Along with Content accessibility, it measures a deliberate decision a site has made about agents rather than something it gets for free.

Checks in this category

  • AI bot rules in robots.txt
    Explicit rules for AI crawlers (GPTBot, ClaudeBot, Google-Extended and others) show that someone has decided who gets in.
  • robots.txt agent-user policy
    User-triggered agents (ChatGPT-User, Claude-User, Perplexity-User) fetch a page because a person asked for it, so blocking them is usually an accident.
  • Emerging checkContent Signals
    Cloudflare Content Signals (Content-Signal in robots.txt) declare how content may be used for search, AI input, and AI training.
  • Emerging checkWeb Bot Auth
    Web Bot Auth lets verified agents prove who they are with HTTP Message Signatures, listed in a signatures directory.