Crawl Budget: What It Is and How to Manage It in 2026

Crawl budget is two numbers multiplied. Google defines it as crawl rate limit multiplied by crawl demand, per Google's official crawl budget documentation (https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget). Both factors move independently.

Crawl rate limit is server-side. Googlebot throttles its request rate when response latency rises, capping concurrency to avoid melting your origin. Per Google's 2025 crawler documentation, latency above 1 second triggers rate reduction (https://developers.google.com/search/docs/crawling-indexing/overview).

Crawl demand reflects interest signals. Googlebot prioritizes URLs that look fresh, popular, or recently updated, and deprioritizes URLs that returned 404 last visit or canonicalize to another URL. Demand shifts daily.

Faceted navigation eats the most budget on retail sites. A single category page exposes 50 to 500 parameter combinations. Googlebot fetches each one until robots.txt tells it to stop.

Session IDs and tracking parameters compound the problem. Each user session generates a unique URL, and Googlebot treats each as a separate fetch target. The duplicate URLs dilute the budget pool.

Soft 404s cost real fetches. A page returning 200 OK with no products found content triggers a fetch, a render, and a quality scoring pass, per Ahrefs' 2025 crawl budget guide (https://ahrefs.com/blog/crawl-budget/). Three wasted steps per request.

Infinite calendar and archive URLs drain the last reserves. A blog with 120 months of archives and 50 tags per post exposes 6,000 archive combinations. Most get fetched once, then re-fetched after small updates.

You open your server logs at 4pm Friday. You grep for Googlebot, Bingbot, and GPTBot user-agent strings, then group requests by URL pattern. The top 20 patterns consume 80% of total bot traffic.

You mark which patterns changed since last week. New URL patterns signal leaked parameters, broken canonical chains, or accidental noindex removals, and each one points to a different upstream cause. The diff is the diagnostic.

You correlate the diff against your last deploy. A staging URL exposed in production, a canonical tag dropped from the head, or a new pagination parameter. The deploy timestamp narrows the search to two hours.

Faceted nav is the most common offender. Per SEMrush's 2025 crawl budget guide, the median audited e-commerce site wastes 20 to 30% of Googlebot requests on parameter combinations that canonicalize elsewhere (https://www.semrush.com/blog/crawl-budget/). The waste scales with catalog size.

Server response time compounds every waste. Googlebot cuts concurrency when TTFB crosses 1 second, per Google's crawler overview (https://developers.google.com/search/docs/crawling-indexing/overview). A 200ms optimization recovers 15% of lost fetches in published case studies.

Sitemap and internal link misalignment wastes the third tier. URLs listed in the XML sitemap but absent from the internal link graph signal low priority to the scheduler, per SEMrush's 2025 crawl budget guide. Sitemap-only URLs crawl less often.

404 chains form a slow leak. Broken internal links pointing to 404s cost a fetch, a 404 response, a recheck on next pass, per Google's crawler documentation. The median crawled site loses 4% of bot requests to 404s.

The math is straightforward. Take your server's Googlebot requests per day from logs, divide by the total URLs Google has indexed for your property in GSC, and you get fetches per indexed URL. Anything below 0.1 signals starvation.

The fix list follows the diagnostic. Robots.txt on faceted parameter combinations, canonical tags on paginated archives, server-side rendering on JavaScript-only content, and 410 Gone on permanently removed URLs. Each cut targets a specific waste class.

Audit quarterly. Crawl patterns drift with templates, query parameters, and content additions, and the budget that served you in March leaks by July if you do not revisit the log files. The calculation is a snapshot, not a constant.