Google-Extended Token: Managing Google's AI Training Crawler

The Google-Extended token specifically governs whether Google can access web content for training its large language models and other AI systems. It does not affect search indexing, ranking, or display in Google Search results. This distinction is critical: blocking Google-Extended preserves search visibility while preventing content from being ingested into AI training pipelines (Google Search Central, 2025).

Google-Extended is implemented as a standard robots.txt user-agent directive. Operators add a dedicated section to their robots.txt file:

User-agent: Google-Extended
Disallow: /

This directive tells Google not to use any content from the site for AI model training. Unlike standard Googlebot directives, Google-Extended applies specifically to AI training use cases and does not interact with search indexing behavior. Google verifies compliance with this directive through its own crawling infrastructure (Google Developers, 2025).

Google-Extended operates as a separate token from Googlebot and Googlebot-Image. A site that blocks Googlebot entirely will also effectively block Google-Extended, but blocking only Google-Extended leaves standard search crawling unaffected. This fine-grained control allows publishers to maintain search presence while opting out of AI training. SEO teams should audit their robots.txt files to ensure Google-Extended is configured as intended (Search Engine Journal, 2025).

Google-Extended applies only to Google's own AI training. It does not affect other AI crawlers, including GPTBot, ClaudeBot, or Common Crawl. Each crawler requires its own robots.txt directives. Additionally, Google-Extended does not retroactively remove previously collected training data. It only prevents future data collection from the point of implementation forward.

Web operators can verify Google-Extended compliance by monitoring server logs for requests from Google IP ranges that match AI training behavior rather than search indexing. Google does not publish a separate user-agent string for Google-Extended requests; instead, compliance is managed through the robots.txt token alone. Traffic analysis tools that categorize crawler types help distinguish AI training requests from standard search crawling.

Review your robots.txt configuration today to confirm Google-Extended directives are properly implemented. Document the specific AI crawlers your site allows and blocks, and verify that Google-Extended rules do not inadvertently conflict with your Googlebot search crawling directives.

Citations: Google Search Central (2025) Google-Extended Token Documentation; Google Developers (2025) Robots.txt Protocol Specification; Search Engine Journal (2025) Managing AI Crawlers with Robots.txt.