Rate limit response headers

RateLimit headers let an agent pace itself instead of discovering your limit by hitting it.

Category: Protocol discovery
Standard: Recommended

1%Top 100 adoption1 of 97 sites implement it 4/9OverviewSpeculativeSpecified, but little evidence anything uses it yet.

Who supports it

CCBotPartial support
Published 25 Sept 2026 · last checked 9 Oct 2026: Common Crawl FAQ'The crawler uses an adaptive back-off algorithm that slows down requests to your website if your web server is responding with a HTTP 429 or 5xx status.' The FAQ does not mention Retry-After or the RateLimit fields. Date is the page's Last-Modified header.

26 others: no evidence yet. See every agent and browser

What it checks

The extension fetches the page being scanned and looks at the response headers for any of these:

RateLimit
RateLimit-Policy
RateLimit-Limit
X-RateLimit-Limit
X-Rate-Limit-Limit
Retry-After

The first two are the fields in the current IETF draft and the third is from an earlier version of it. The X- names are the long-standing convention, and Retry-After is the standard HTTP field (RFC 9110). Any one of them is enough to pass.

Results

Status	When
Pass	At least one of the headers is on the response
N/A	None of the headers is on the response

How to fix

On API responses, send the IETF draft fields so an agent knows its quota and how much of it is left:

RateLimit-Policy: "default";q=100;w=60
RateLimit: "default";r=42;t=18

When you reject a request for going over the limit, answer 429 Too Many Requests with a Retry-After header giving the number of seconds to wait.

The check reads the URL you scan, so scan an API endpoint to test your API’s headers rather than a marketing page.

Support by agent and browser

Announced means the vendor’s own docs or announcement say so. Published is the date on the vendor’s own page, where it has one. Checked is the last time our weekly research found that page still saying so.

Agent	Status	Source
AI crawlers
CCBot Common Crawl	Partial support	partial Common Crawl FAQ Published 25 Sept 2026 · Checked 9 Oct 2026'The crawler uses an adaptive back-off algorithm that slows down requests to your website if your web server is responding with a HTTP 429 or 5xx status.' The FAQ does not mention Retry-After or the RateLimit fields. Date is the page's Last-Modified header.

No evidence yet26

ChatGPT
Claude
Gemini
Perplexity
Copilot
Meta AI
ChatGPT Atlas
Comet
Dia
Gemini in Chrome
Edge Copilot Mode
Claude Code
Cursor
Codex
GitHub Copilot
Windsurf
Applebot
Amazonbot
Bytespider
Cohere
Mistral
DuckAssist
Chrome
Edge
Firefox
Safari

Names and marks belong to their owners and are used only to identify products. None of these vendors is affiliated with Sitebulb. Spotted something out of date? Send a correction.

Adoption on the top 100 sites

1 of 97 measured sites implement it

Scanned 23 Oct 2026 with the Sitebulb Labs extension, against the Tranco list (tranco-56Z8N). 3 sites couldn’t be measured (bot challenge or timeout) and are left out of the share.

9 Oct 2026 1% → 23 Oct 2026 1%

Sites implementing it1

#75 roblox.com

Overview

4/9

Speculative

Specified, but little evidence anything uses it yet.

How this rating is worked out4 of 9 points

Factor	Points	Why
Vendor support	1 / 3	0 vendors supporting, 1 partly or planned
Top-100 adoption	1 / 3	1% of measured sites
Spec maturity	2 / 3	Standards-track draft

8 or more points is Established, 5 to 7 is Gaining traction, 2 to 4 is Speculative and under 2 is Stalled. A feature is also Stalled when more vendors have said no than yes. Read the spec.

Community insights

No community insights for this feature yet.

Quotes are from public posts, checked against the original before they appear here. Suggest an insight, or ask for yours to be removed.