robots.txt present & valid
A reachable robots.txt lets agents discover crawl rules and sitemaps.
- Category
- Discoverability
- Standard
- Established
What it checks
The extension fetches /robots.txt from the site’s origin and parses it. It
passes when the file is reachable and contains at least one User-agent group
or Sitemap line. An agent reads this file first to learn what it may crawl
and where the sitemap lives.
Results
| Status | When |
|---|---|
| Pass | robots.txt is reachable and has at least one User-agent group or Sitemap line |
| Warn | The file is reachable but has no User-agent groups or Sitemap lines |
| Fail | No robots.txt was found (a 404, another error, or no response) |
How to fix
Serve a plain-text file at /robots.txt on the site root. A minimal one that
allows everything and points to the sitemap:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlSeveral other checks read the same file, so getting it right helps them too: Sitemap available, AI bot rules in robots.txt, and robots.txt agent-user policy.