Blocking AI bots in robots.txt: The Complete 2026 Guide

A robots.txt file lives at the root of your domain. It uses simple directives. Each crawler gets its own block.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

This blocks all major AI training crawlers. Place this at the top of your file. Google's robots.txt specification is the authoritative reference for syntax rules.

The AI crawler list grows every quarter. Here is the full set of user-agent tokens to block in 2026.

GPTBot
ClaudeBot
Google-Extended
Google-CloudVertexBot
Google-Experimental
ChatGPT-User
PerplexityBot
CCBot (Common Crawl)
Applebot-Extended
Meta-ExternalAgent
Bytespider
cohere-ai
omgili
omgilibot
ImagesiftBot
Amazonbot

Not all of these are training crawlers. Some are data aggregators. Blocking them depends on your goals. AI content crawler aggregation community lists track the full set monthly.

Robots.txt does not rate limit. It controls access. You can say yes or no. You cannot say "visit once per minute." The Crawl-delay directive exists but many AI crawlers ignore it. The robots.txt standard does not enforce rate limiting.

User-agent: GPTBot
Disallow: /private/
Crawl-delay: 10

GPTBot respects Crawl-delay. Other crawlers may not. The robots.txt exclusion standard (RFC 9309) defines Crawl-delay as optional. Compliance varies. Check each crawler's documentation.

You can block AI crawlers from specific sections while leaving public content accessible.

User-agent: GPTBot
Disallow: /blog/
Allow: /blog/about-us$

The Allow directive overrides Disallow for matching URLs. Google introduced this extension. Most crawlers support it. Use it to carve out specific pages from a broad block. Google's robots.txt guidance explains the precedence rules.

Any crawler can lie about its user-agent. Robots.txt stops honest crawlers only. Malicious scrapers ignore it. Recent data from Imperva shows that 30% of AI crawler traffic comes from unverified agents. Imperva's 2025 Bad Bot Report found that spoofed AI agents are the fastest growing bot category.

Use server-level blocking as a second layer. Block by IP range for known AI crawlers. Block by behavior pattern for unknown crawlers. Rate limit aggressive request patterns at the proxy level. Robots.txt is your first defense. It is not your only defense.

Content Delivery Networks provide AI bot blocking as a feature. Cloudflare offers "Block AI Crawlers" as a single toggle. It applies the complete AI bot blocklist at the edge. Cloudflare's AI bot blocking covers GPTBot, ClaudeBot, Google-Extended, and others. The toggle updates automatically as new crawlers appear.

Akamai, Fastly, and AWS CloudFront offer similar features. Check their bot management documentation. A WAF rule with user-agent pattern matching adds another layer. Block AI crawlers before they reach your origin server.

Robots.txt blocks are not instant. Crawlers read the file at the start of a crawl session. A crawler that started before your update may continue for hours. Monitor your logs after making changes. Confirm that blocked crawler traffic actually stops.

Check response codes. A blocked crawler should receive 403 or 404 responses depending on your configuration. If you still see 200 responses from a blocked user-agent, your robots.txt syntax may be wrong. Double-check the user-agent string. A trailing space or missing line break invalidates the directive. The robots.txt validator from Google can test your file syntax.

AI companies are adding more crawlers. Google added Google-CloudVertexBot and Google-Experimental in 2025. OpenAI may add new crawlers for future models. Meta has Meta-ExternalAgent. The pattern is clear: each model release brings a new crawler.

Subscribe to AI bot change feeds. The ai-robots-txt community repository on GitHub tracks additions. Cloudflare's bot list updates automatically. Set a calendar reminder to review your robots.txt every month.

The blocking AI bots in robots.txt audit. Note the gap between robots.txt and server-level enforcement. Robots.txt stops honest crawlers. Server rules stop impostors. You need both layers. Audit quarterly.