---
title: "robots.txt agent-user policy · Sitebulb Labs"
description: "User-triggered agents (ChatGPT-User, Claude-User, Perplexity-User) fetch a page because a person asked for it, so blocking them is usually an accident."
url: https://labs.sitebulb.com/docs/checks/bot-access-control/robots-agent-user-policy/
---

# robots.txt agent-user policy

User-triggered agents (ChatGPT-User, Claude-User, Perplexity-User) fetch a page because a person asked for it, so blocking them is usually an accident.

- Category

  [Bot access control](https://labs.sitebulb.com/docs/checks/bot-access-control)

- Standard

  Established

## What it checks

Some agents only visit a page when a person asks them to: someone pastes a link into ChatGPT, Claude, or Perplexity and asks about it. The extension works out which `robots.txt` group applies to each of **ChatGPT-User**, **Claude-User**, and **Perplexity-User**, and whether that group blocks the whole site.

It also tells a deliberate block apart from collateral damage. A blanket `User-agent: *` / `Disallow: /` catches these agents too, and most sites that do it did not mean to.

A site with no `robots.txt` blocks nobody, so it passes here. The missing file is reported by [robots.txt present & valid](https://labs.sitebulb.com/docs/checks/discoverability/robots-txt) instead, so it is not counted twice.

## Results

| Status   | When                                                                         |
| -------- | ---------------------------------------------------------------------------- |
| **Pass** | None of the three agents is blocked, or there is no `robots.txt`             |
| **Fail** | One or more of them is blocked, either by name or by a blanket wildcard rule |

## How to fix

Allow the user-triggered agents explicitly. Blocking them takes your site out of answers people ask for, and does nothing to keep it out of training data, which uses different crawlers.

```txt
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /
```

If your `*` group disallows everything, a named group like this one overrides it for these agents.

To control training use, write rules for the training crawlers instead. See [AI bot rules in robots.txt](https://labs.sitebulb.com/docs/checks/bot-access-control/ai-bot-rules).