Discoverability

Learn how the extension checks sitemaps, robots.txt, redirects and discovery hints that help AI agents find pages and services on your site.

Before an agent can read anything, it has to find it. These checks look at the files and responses that tell an agent what a site contains and where to go: robots.txt, the sitemap, HTTP Link headers, honest 404s, and the newer catalogs that list a domain’s agents, APIs, and MCP servers.

Discoverability carries a weight of 1 in the overall score. A well-run site passes most of it without doing anything agent-specific, so it counts for less than Content accessibility and Bot access control.

Checks in this category

  • robots.txt present & valid
    A reachable robots.txt lets agents discover crawl rules and sitemaps.
  • Sitemap available
    A sitemap gives agents a list of the pages worth reading or indexing.
  • Link response headers
    HTTP Link headers (such as rel="alternate", "canonical" or "describedby") give agents structured discovery hints.
  • Agent-friendly 404s
    A missing page must answer 404 or 410. Returning a 200 tells an agent the page exists.
  • Redirect hygiene
    Agents that don't run JavaScript never follow a meta refresh or a location.href hop, so they only see the initial page.
  • Sitemap freshness (lastmod)
    <lastmod> lets an agent re-read only what changed instead of recrawling everything.
  • Emerging checkDNS for AI Discovery (DNS-AID)
    A DNS TXT record listing agent endpoints lets agents find services at the DNS layer.
  • Emerging checkARD catalog (ai-catalog)
    An Agentic Resource Discovery catalog lets agents list the MCP servers, agents, skills and APIs a domain publishes.
  • Emerging checkARD entry validity
    Every ARD entry must carry a domain-anchored urn:air identifier, a display name, a media type, and exactly one of url or data.
  • Emerging checkARD trust manifest
    A trustManifest binds catalog entries to a verifiable identity so clients can check who published them.
  • Emerging checkNLWeb Schema Feeds
    A schemamap: directive in robots.txt points agents at machine-readable feeds of your structured data.