Robots.txt for AI crawlers: The Complete 2026 Guide

The robots.txt protocol, standardized in RFC 9309 since 2022, has become a frontline tool for managing AI crawler access. As of 2026, dozens of AI-specific user agents crawl the web daily, and every site operator needs a deliberate strategy for allowing, restricting, or blocking each one. The old approach of a generic Disallow directive for all bots no longer suffices.

Understanding who is crawling your site is the first step. The most important AI crawler user agents as of 2026:

• GPTBot (Mozilla/5.0 compatible; GPTBot/1.0) -- OpenAI's training data crawler. Also watch for ChatGPT-User which powers ChatGPT's real-time browsing feature. These serve different purposes and should be managed separately.
• Google-Extended (Googlebot-Extended) -- Google's AI training crawler, distinct from Googlebot used for search indexing. Launched in 2023, it allows site owners to opt out of AI training while keeping search index access.
• ClaudeBot (Anthropic; Mozilla/5.0 compatible; ClaudeBot/1.0) -- Anthropic's web crawler for Claude training data and real-time search.
• PerplexityBot (Perplexity/1.0) -- Perplexity AI's crawler, used both for training and live answers.
• Cohere's crawler and Meta's crawler (Meta-ExternalAgent) also appear in server logs with increasing frequency.

The robots.txt file lives at the root of your domain (e.g., https://example.com/robots.txt). For AI crawler management, you have three approaches:

ChatGPT-User is the user agent used when a ChatGPT user asks for real-time web results. Blocking GPTBot (training) does not block ChatGPT-User. To restrict both:

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

This matters because real-time browsing generates page impressions and referral traffic. Many sites choose to allow ChatGPT-User while blocking GPTBot.

Robots.txt is a voluntary protocol. Not all AI crawlers honor it equally. Anthropic and OpenAI have publicly committed to respecting robots.txt directives, but independent audits in 2025 showed varying compliance rates. Server-side access controls, IP rate limiting, and authentication provide stronger guarantees for sensitive content.

Additionally, robots.txt does not grant you direct control over how crawled content is used. It controls access, not training rights. For usage restrictions, you need Terms of Service enforcement or technical measures like paywalls and authentication.

AI crawlers can generate significant traffic. Monitor your access logs for excessive requests from specific AI user agents and use crawl-delay directives:

User-agent: GPTBot
Crawl-Delay: 10
Allow: /

This tells the crawler to wait 10 seconds between requests. Not all crawlers honor Crawl-Delay, but most do. Combine this with server-side rate limiting as a hard backstop.

• [ ] List every AI crawler user agent appearing in your server logs from the last 30 days
• [ ] Verify your robots.txt includes explicit directives for GPTBot, Google-Extended, ClaudeBot, and PerplexityBot
• [ ] Add separate ChatGPT-User directive if you want to distinguish training from real-time browsing
• [ ] Set appropriate Crawl-Delay values if AI crawler traffic is impacting server performance
• [ ] Test your robots.txt with Google's Robots Testing Tool and manual curl validation
• [ ] Confirm that blocking directives use full paths and do not accidentally block critical resources like CSS or JS files
• [ ] Review your policy quarterly -- new AI crawlers appear frequently and your robots.txt should evolve

Robots.txt is your first line of defense and invitation for AI crawlers. A deliberate, up-to-date configuration ensures you control who accesses your content, at what rate, and under what terms. Treat your robots.txt as a living policy document, not a static file you set once and forget.

Citations

1. RFC 9309. "Robots Exclusion Protocol." IETF, 2022. https://www.rfc-editor.org/rfc/rfc9309
2. Google Search Central. "About Google-Extended." Google Developers, 2024. https://developers.google.com/search/docs/crawling-indexing/google-extended
3. OpenAI. "GPTBot and ChatGPT-User Documentation." OpenAI Platform, 2025. https://platform.openai.com/docs/gptbot
4. Anthropic. "ClaudeBot and Web Crawling." Anthropic Documentation, 2025. https://docs.anthropic.com/en/docs/claudebot