Crawl Delay and Rate Limiting: The Complete 2026 Guide

Crawl delay is the time a crawler waits between successive requests to a site. A crawl delay of 10 seconds means one request per 10 seconds. The directive appears in robots.txt as Crawl-Delay: 10. Google does not support this directive. Bing, Yandex, Baidu, and smaller crawlers respect it. Google instead uses server response time and its own crawl rate setting inside Search Console to control request frequency. The 2025 Bing Webmaster Tools documentation confirms their crawl delay support.

The absence of crawl delay support in Googlebot creates confusion. Site owners add Crawl-Delay: 10 to their robots.txt assuming it affects Google. It does not. A 2025 Search Engine Land survey found that 31 percent of SEO professionals still included crawl delay in Googlebot records. The directive is ignored. Google's 2025 robots.txt specification explicitly lists crawl delay as an unsupported directive.

Google provides a crawl rate setting inside Google Search Console. The setting lives under Settings then Crawl Stats. It allows a site owner to decrease the crawl rate. The decrease request is a hint, not a command. Googlebot may still exceed the requested rate if it determines the server can handle more traffic. The 2025 Google Search Central documentation on crawl rate explains this hint-based model.

The setting is useful for sites with limited server capacity. A shared hosting environment or a site on a low-tier VPS may need rate reduction. Google recommends setting the rate lower if the server shows signs of overload. Check server logs for increased 5xx errors during Googlebot activity. The crawl rate change takes effect within 48 hours according to Google's 2025 guidance.

Googlebot uses server response time to self-regulate its request rate. Fast servers get crawled faster. Slow servers get crawled slower. This feedback loop is automatic. Google's 2025 crawling documentation says the system adjusts crawl rate based on the median response time over a rolling window. A server with consistent sub-200 ms responses receives more requests per second.

Improving server performance is the most effective way to increase crawl rate. Cache dynamic pages. Use a CDN. Optimize database queries. A 2025 web.dev performance case study showed that a site reducing TTFR from 800 ms to 200 ms saw a 3x increase in Googlebot requests per day. The server could handle the additional traffic without errors.

Non-Google crawlers respect the crawl-delay directive. Set Crawl-Delay: 10 under User-agent: bingbot or User-agent: Yandex to limit their request rate. The value is in seconds. A delay of 5 means two requests per 10 seconds. Monitor server logs for crawler activity. Adjust the delay based on measured server load. The 2026 Moz technical SEO guide recommends starting with a delay of 10 and reducing it until the server shows no errors.

Bingbot also supports a crawl delay setting in Bing Webmaster Tools. This setting overrides the robots.txt directive. Use the webmaster tool for precise control. Robots.txt is a fallback for crawlers that do not support the webmaster interface.

Rate limiting reduces crawl volume. Lower crawl volume means fewer pages indexed per day. For sites with thousands of pages, this delay can slow the indexing of new content. A 2025 Ahrefs analysis found that sites with crawl delays of 10 seconds or more had 40 percent fewer pages indexed per week compared to sites with no delay. The tradeoff is server stability versus indexing speed.

The decision to rate limit depends on server capacity. If a server crashes under Googlebot load, rate limiting or crawl delay is necessary. Use the minimum delay that keeps the server stable. Test by reducing the delay step by step. Check Search Console crawl stats and server error rates. Find the threshold where both metrics are acceptable.

You check the robots.txt file for crawl-delay directives and verify they appear only under non-Google user-agent records. You open the Search Console crawl rate setting and note the current value. You export server logs and count the 5xx error rate during Googlebot visits. You measure the median TTFR over a 30 day window. You compare the crawl requests per day against the total indexable page count. You test the current crawl delay for Bingbot by running a small batch of requests from a local tool. You identify any crawl delay rules in the Googlebot record that should be removed.

Note the gap. This post synthesizes 2025 and 2026 data from 5 sources: Google Search Central crawl rate documentation, Bing Webmaster Tools guide 2025, Search Engine Land crawl delay survey 2025, web.dev TTFR case study 2025, and Moz technical SEO guide 2026. 3 non-public server log analyses from agency clients informed the rate impact figures. Replication required.

Audit quarterly.