Identifying ClaudeBot in logs: The Complete 2026 Guide

Anthropic documents a single official crawler. The user-agent string is:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible with ClaudeBot/1.0; +https://www.anthropic.com/claude-bot

The markers are ClaudeBot/1.0 and the verification URL https://www.anthropic.com/claude-bot. There is no separate user-agent for Claude's chat product. Anthropic routes all external crawl traffic through ClaudeBot. Anthropic's crawler documentation lists the official specifications.

Anthropic publishes its IP address ranges. You can find them in the official documentation. Anthropic's IP ranges for ClaudeBot include both IPv4 and IPv6 prefixes. The IPs change periodically as Anthropic scales its infrastructure.

Run reverse DNS on any IP that claims to be ClaudeBot. Verified IPs resolve to hostnames under anthropic.com. Unverified IPs may be impostors. Any IP that does not resolve to an anthropic.com host should be treated with suspicion.

Grep is the fastest method. Search for the ClaudeBot token.

grep ClaudeBot /var/log/nginx/access.log

ClaudeBot variants include Claude-Web in some older logs. Anthropic deprecated that agent in 2025. The current agent is always ClaudeBot/1.0. Filter out anything that uses Claude without the /1.0 version suffix. It is likely a fake.

Count requests by hour.

grep ClaudeBot /var/log/nginx/access.log | awk '{print $4}' | cut -d: -f1,2 | sort | uniq -c | sort -rn

For JSON structured logs, use jq.

cat access.json | jq 'select(.http_user_agent | test("ClaudeBot"))'

Datadog's log management supports agent filtering with grok parsers. You can set up a dedicated facet for ClaudeBot.

ClaudeBot is aggressive. It hits your site in bursts. Anthropic runs large-scale training data collection. The crawler sends requests from many IPs in parallel. Traffic spikes are common.

ClaudeBot respects robots.txt but interprets Allow and Disallow strictly. A misconfigured robots.txt can result in unintended crawl volume. Check your directives carefully. Google's robots.txt documentation has the standard. The same standard applies to all AI crawlers.

The Accept header is */*. The Accept-Language header varies. ClaudeBot does not include a Referer header in most requests. These header patterns help distinguish real ClaudeBot traffic from scrapers.

ClaudeBot is one of the top three AI crawlers by request volume. A 2025 study by Imperva showed that AI crawler traffic grew 850% year-over-year. Imperva's Bad Bot Report 2025 ranked ClaudeBot in the top five bots across all categories. Cloudflare's 2025 radar data confirms similar trends. Cloudflare Radar shows ClaudeBot as a leading crawler on content-heavy sites.

Sites in the publishing, education, and documentation sectors see the highest ClaudeBot volume. A single crawl session can run tens of thousands of requests.

You can block ClaudeBot in robots.txt.

User-agent: ClaudeBot
Disallow: /

Rate limiting at the reverse proxy is more surgical. Use nginx limit_req zones keyed on the user-agent string. This preserves access for legitimate users while controlling Anthropic's crawl rate. Anthropic's data controls FAQ explains how site owners can manage their data collection preferences.

GoAccess can classify ClaudeBot if you add a custom parsing rule. The elasticsearch ingest pipeline supports user-agent pattern matching. Logstash grok filters can label ClaudeBot traffic. The commercial tool Botify includes ClaudeBot in its crawler database.

The ClaudeBot audit. Note the gap between ClaudeBot and other AI crawlers in burst behavior. Anthropic crawls in intense, short-lived waves. Traditional bots crawl steadily. Your rate limiting strategy must account for this difference. Audit quarterly.