Robots.txt for AI Crawlers: A Complete Configuration Guide

Robots.txt remains the primary mechanism for controlling AI crawler access to web content. With the proliferation of AI training bots from OpenAI, Anthropic, Google, Meta, Apple, Amazon, and others, configuring robots.txt correctly requires understanding dedicated user-agent tokens, directive...

AI-Specific User-Agent Tokens

Major AI companies now support dedicated robots.txt tokens for AI training access control:

Google-Extended: Controls Google AI model training (Gemini, SGE)
GPTBot: Controls OpenAI training data collection
ClaudeBot: Controls Anthropic model training
Applebot-Extended: Controls Apple Intelligence training
Meta-ExternalAgent: Controls Meta AI training
Amazonbot: Controls Amazon AI training
CCBot: Controls Common Crawl data collection (Google Developers, 2025)

Recommended Configuration Structure

Organize your robots.txt with AI crawler directives grouped and preceding general crawler rules:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: *
Allow: /

This structure blocks all known AI training crawlers while allowing search engines and other legitimate bots full access. The wildcard rule at the bottom catches any crawlers not explicitly addressed (Cloudflare, 2025).

Specificity and Precedence

Robots.txt matching follows longest-prefix precedence. More specific paths take priority over less specific ones. If you want to allow AI crawlers on certain sections while blocking others, place the more specific allow rules before the general disallow:

User-agent: GPTBot
Allow: /public/
Disallow: /

This configuration restricts GPTBot to the public directory only. Always test your robots.txt configuration using validation tools to ensure directives are interpreted as intended (Search Engine Journal, 2025).

Limitations of Robots.txt for AI Crawlers

Robots.txt is a voluntary protocol. Not all AI crawlers respect it, and those that do may have varying implementation quality. Some crawlers may ignore robots.txt for user-initiated requests (like GPTBot-User). Additionally, robots.txt does not prevent content from being used through secondary sources such as RSS feeds, APIs, or user-shared links. For comprehensive protection, combine robots.txt with server-level access controls and terms of service.

Testing and Validation

Use Google's robots.txt testing tool, and supplement with direct HTTP requests using curl to verify your AI crawler directives. Monitor access logs after configuration changes to confirm that blocked AI crawlers stop requesting pages. Automated testing should be part of your deployment pipeline for robots.txt changes.

Audit your robots.txt file today against this recommended AI crawler configuration. Add or update directives for any missing AI crawler tokens. Test your configuration with validation tools and monitor access logs for 48 hours to confirm the directives are being respected.

Citations: Google Developers (2025) Robots.txt Protocol Specification; Cloudflare (2025) AI Crawler Management Guide; Search Engine Journal (2025) Managing AI Crawlers with Robots.txt.