Get now

Discoverability

Learn how the extension checks sitemaps, robots.txt, redirects and discovery hints that help AI agents find pages and services on your site.

Before an agent can read anything, it has to find it. These checks look at the files and responses that tell an agent what a site contains and where to go: robots.txt, the sitemap, HTTP Link headers, honest 404s, and the newer catalogs that list a domain’s agents, APIs, and MCP servers.

Discoverability carries a weight of 1 in the overall score. A well-run site passes most of it without doing anything agent-specific, so it counts for less than Content accessibility and Bot access control.

Checks in this category

  • robots.txt present & valid
    A reachable robots.txt lets agents discover crawl rules and sitemaps. Top 100 adoption 87%
  • Sitemap available
    A sitemap gives agents a list of the pages worth reading or indexing. Top 100 adoption 65%
  • Link response headers
    HTTP Link headers (such as rel="alternate", "canonical" or "describedby") give agents structured discovery hints. Top 100 adoption 10%
  • Agent-friendly 404s
    A missing page must answer 404 or 410. Returning a 200 tells an agent the page exists. Top 100 adoption 86%
  • Redirect hygiene
    Agents that don't run JavaScript never follow a location.href hop, and few document following a meta refresh, so they may only see the initial page. Top 100 adoption 75%
  • Sitemap freshness (lastmod)
    <lastmod> lets an agent re-read only what changed instead of recrawling everything. Top 100 adoption 52%
  • Emerging checkDNS for AI Discovery (DNS-AID)
    A DNS TXT record listing agent endpoints lets agents find services at the DNS layer. Top 100 adoption 1%
  • Emerging checkARD catalog (ai-catalog)
    An Agentic Resource Discovery catalog lets agents list the MCP servers, agents, skills and APIs a domain publishes. Top 100 adoption 1%
  • Emerging checkARD entry validity
    Every ARD entry must carry a domain-anchored urn:air identifier, a display name, a media type, and exactly one of url or data. Top 100 adoption 1%
  • Emerging checkARD trust manifest
    A trustManifest binds catalog entries to a verifiable identity so clients can check who published them. Top 100 adoption 0%
  • Emerging checkNLWeb Schema Feeds
    A schemamap: directive in robots.txt points agents at machine-readable feeds of your structured data. Top 100 adoption 0%