---
title: "robots.txt present & valid · Sitebulb Labs"
description: "A reachable robots.txt lets agents discover crawl rules and sitemaps."
url: https://labs.sitebulb.com/docs/checks/discoverability/robots-txt/
---

# robots.txt present & valid

A reachable robots.txt lets agents discover crawl rules and sitemaps.

- Category

  [Discoverability](https://labs.sitebulb.com/docs/checks/discoverability)

- Standard

  Established

## What it checks

The extension fetches `/robots.txt` from the site’s origin and parses it. It passes when the file is reachable and contains at least one `User-agent` group or `Sitemap` line. An agent reads this file first to learn what it may crawl and where the sitemap lives.

## Results

| Status   | When                                                                                |
| -------- | ----------------------------------------------------------------------------------- |
| **Pass** | `robots.txt` is reachable and has at least one `User-agent` group or `Sitemap` line |
| **Warn** | The file is reachable but has no `User-agent` groups or `Sitemap` lines             |
| **Fail** | No `robots.txt` was found (a 404, another error, or no response)                    |

## How to fix

Serve a plain-text file at `/robots.txt` on the site root. A minimal one that allows everything and points to the sitemap:

```txt
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
```

Several other checks read the same file, so getting it right helps them too: [Sitemap available](https://labs.sitebulb.com/docs/checks/discoverability/sitemap), [AI bot rules in robots.txt](https://labs.sitebulb.com/docs/checks/bot-access-control/ai-bot-rules), and [robots.txt agent-user policy](https://labs.sitebulb.com/docs/checks/bot-access-control/robots-agent-user-policy).