robots.txt agent-user policy
User-triggered agents (ChatGPT-User, Claude-User, Perplexity-User) fetch a page because a person asked for it, so blocking them is usually an accident.
- Category
- Bot access control
- Standard
- Established
What it checks
Some agents only visit a page when a person asks them to: someone pastes a link
into ChatGPT, Claude, or Perplexity and asks about it. The extension works out
which robots.txt group applies to each of ChatGPT-User, Claude-User,
and Perplexity-User, and whether that group blocks the whole site.
It also tells a deliberate block apart from collateral damage. A blanket
User-agent: * / Disallow: / catches these agents too, and most sites that
do it did not mean to.
A site with no robots.txt blocks nobody, so it passes here. The missing file
is reported by robots.txt present & valid
instead, so it is not counted twice.
Results
| Status | When |
|---|---|
| Pass | None of the three agents is blocked, or there is no robots.txt |
| Fail | One or more of them is blocked, either by name or by a blanket wildcard rule |
How to fix
Allow the user-triggered agents explicitly. Blocking them takes your site out of answers people ask for, and does nothing to keep it out of training data, which uses different crawlers.
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /If your * group disallows everything, a named group like this one overrides
it for these agents.
To control training use, write rules for the training crawlers instead. See AI bot rules in robots.txt.