AI crawler traffic patterns: The Complete 2026 Guide

AI crawlers cluster requests. They do not spread them evenly. ClaudeBot is the most extreme example. It hits a site in short, intense waves. A burst can deliver 10,000 requests in 30 minutes. Then silence for hours. This pattern differs from Googlebot, which crawls at a steady rate. Cloudflare's 2025 AI bot traffic analysis documented these bursts across thousands of sites.

Server provisioning must handle these spikes. A site that normally serves 100 requests per minute can see 500 AI crawler requests per minute during a burst.

There are two distinct AI crawler categories. Training crawlers (GPTBot, ClaudeBot, Google-Extended) collect large datasets for model training. Live-query crawlers (ChatGPT-User, PerplexityBot) fetch pages in response to a user request.

Training crawlers make many requests per session. They follow internal links aggressively. They return to a site every few weeks. Live-query crawlers make one or two requests per session. They hit a site sporadically. Perplexity's bot documentation explains their one-page-per-query approach.

The traffic patterns reflect this difference. Training crawlers produce high volume in compressed time windows. Live-query crawlers produce low volume with no predictable pattern.

AI crawlers use high concurrency. ClaudeBot opens many simultaneous connections. Google-Extended also uses multiple concurrent connections. GPTBot limits itself to one concurrent connection by default. OpenAI states this in their GPTBot documentation.

Traditional search bots typically use 5-20 concurrent connections. AI training crawlers can use 50-100 or more. This strains connection pools and file descriptor limits. Monitor your netstat and connection counts. NGINX connection limiting can protect your upstream servers from excessive concurrent crawler connections.

AI training crawlers follow links to significant depth. They want the full graph of your site. GPTBot and ClaudeBot follow pagination, category links, and archive pages. They do not stop at the first three levels. Traditional search bots also follow links, but AI crawlers are more aggressive about reaching every accessible URL.

Check your log for crawlers hitting old content. AI crawlers visit pages from 2010 that Googlebot has not touched in years. They are data-hungry. Any public URL is a target. Jakub Kostecki's AI crawler analysis showed AI crawlers hitting 3x more unique URLs per session than Googlebot.

AI crawlers show less time-of-day variation than human traffic. They run jobs at all hours. The bots are geographically distributed. Crawl traffic fills your off-peak hours. This is useful for capacity planning. Your servers stay warm at 3 AM since AI crawlers are active.

Some operators shift AI crawler traffic to low-priority worker pools. This protects user-facing response times. Kubernetes pod autoscaling can handle the extra load if configured properly. Honeycomb's observability guide shows how to distinguish bot patterns from user patterns in telemetry data.

Traffic patterns change when you set robots.txt rules. A Disallow: / directive for a specific crawler drops their traffic to zero. A Crawl-delay directive spreads their requests over a longer period. This transforms a burst pattern into a trickle.

Rate limiting at the proxy level changes traffic shape. An nginx limit_req zone with burst=20 smooths out ClaudeBot bursts. The crawler backs off and retries, spreading requests across hours instead of minutes.

AI crawler traffic correlates with model training cycles. When a company trains a new model, crawl volume spikes. Early 2025 saw a large GPTBot spike during GPT-5 training. Mid-2025 saw a ClaudeBot spike during Claude 4 training. Late 2025 saw Google-Extended spikes during Gemini 2.0 training.

Watch the AI news cycle. A model announcement usually means a crawl wave is coming. Simon Willison's blog documents his direct experience with these patterns.

Measure AI crawler traffic as a percentage of total requests. Track the ratio over time. Use a weekly dashboard. Compare AI crawler traffic to search engine bot traffic. A healthy site sees AI crawlers at 5-15% of total requests. An AI-heavy site may see 30% or more.

Track these metrics per crawler. GPTBot requests per day. ClaudeBot requests per day. Google-Extended requests per day. Compare week-over-week and month-over-month.

The AI crawler traffic patterns audit. Note the gap between training crawler bursts and live-query crawler trickles. Your infrastructure must handle both patterns. The burst pattern is the harder problem. Audit quarterly.