---
title: "Discoverability · Sitebulb Labs"
description: "Learn how the extension checks sitemaps, robots.txt, redirects and discovery hints that help AI agents find pages and services on your site."
url: https://labs.sitebulb.com/docs/checks/discoverability/
---

# Discoverability

Learn how the extension checks sitemaps, robots.txt, redirects and discovery hints that help AI agents find pages and services on your site.

Before an agent can read anything, it has to find it. These checks look at the files and responses that tell an agent what a site contains and where to go: `robots.txt`, the sitemap, HTTP `Link` headers, honest 404s, and the newer catalogs that list a domain’s agents, APIs, and MCP servers.

Discoverability carries a **weight of 1** in the overall score. A well-run site passes most of it without doing anything agent-specific, so it counts for less than [Content accessibility](https://labs.sitebulb.com/docs/checks/content-accessibility) and [Bot access control](https://labs.sitebulb.com/docs/checks/bot-access-control).

## Checks in this category

- [**robots.txt present & valid**](https://labs.sitebulb.com/docs/checks/discoverability/robots-txt)\
  A reachable robots.txt lets agents discover crawl rules and sitemaps.
- [**Sitemap available**](https://labs.sitebulb.com/docs/checks/discoverability/sitemap)\
  A sitemap gives agents a list of the pages worth reading or indexing.
- [**Link response headers**](https://labs.sitebulb.com/docs/checks/discoverability/link-headers)\
  HTTP Link headers (such as rel="alternate", "canonical" or "describedby") give agents structured discovery hints.
- [**Agent-friendly 404s**](https://labs.sitebulb.com/docs/checks/discoverability/agent-friendly-404)\
  A missing page must answer 404 or 410. Returning a 200 tells an agent the page exists.
- [**Redirect hygiene**](https://labs.sitebulb.com/docs/checks/discoverability/redirect-hygiene)\
  Agents that don't run JavaScript never follow a meta refresh or a location.href hop, so they only see the initial page.
- [**Sitemap freshness (lastmod)**](https://labs.sitebulb.com/docs/checks/discoverability/sitemap-lastmod)\
  \<lastmod> lets an agent re-read only what changed instead of recrawling everything.
- Emerging check[**DNS for AI Discovery (DNS-AID)**](https://labs.sitebulb.com/docs/checks/discoverability/dns-aid)\
  A DNS TXT record listing agent endpoints lets agents find services at the DNS layer.
- Emerging check[**ARD catalog (ai-catalog)**](https://labs.sitebulb.com/docs/checks/discoverability/ard-catalog)\
  An Agentic Resource Discovery catalog lets agents list the MCP servers, agents, skills and APIs a domain publishes.
- Emerging check[**ARD entry validity**](https://labs.sitebulb.com/docs/checks/discoverability/ard-entries-valid)\
  Every ARD entry must carry a domain-anchored urn:air identifier, a display name, a media type, and exactly one of url or data.
- Emerging check[**ARD trust manifest**](https://labs.sitebulb.com/docs/checks/discoverability/ard-trust-manifest)\
  A trustManifest binds catalog entries to a verifiable identity so clients can check who published them.
- Emerging check[**NLWeb Schema Feeds**](https://labs.sitebulb.com/docs/checks/discoverability/nlweb-schema-feeds)\
  A schemamap: directive in robots.txt points agents at machine-readable feeds of your structured data.