AI Crawler Server-Level Blocking: Nginx and Apache Configuration Guide

Block AI crawlers in Nginx using the map module to create a block list:

map $http_user_agent $ai_crawler_block {
    default 0;
    ~*GPTBot 1;
    ~*ClaudeBot 1;
    ~*CCBot 1;
    ~*Meta-ExternalAgent 1;
    ~*Amazonbot 1;
    ~*Bytespider 1;
    ~*PerplexityBot 1;
}

server {
    if ($ai_crawler_block) {
        return 403;
    }
}

The map module approach is more efficient than using multiple if statements because Nginx evaluates the map once per request. Return 403 (Forbidden) to block with a clear signal, or 444 (Connection Close) for silent dropping (Nginx, 2025).

Supplement user-agent blocking with IP range blocks using the geo module:

geo $ai_crawler_ip {
    default 0;
    include /etc/nginx/ai-crawler-ips.conf;
}

server {
    if ($ai_crawler_ip) {
        return 403;
    }
}

Maintain your IP range list in an external file sourced from the DNS TXT records published by AI companies. Automate the IP list updates with a cron job that queries _openai, _anthropic, and other DNS records weekly (Cloudflare, 2025).

For Apache environments, use mod_rewrite to block AI crawlers:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} GPTBot|ClaudeBot|CCBot|Meta-ExternalAgent [NC]
RewriteRule ^ - [F,L]

The F flag returns a 403 Forbidden response. Place these rules early in your configuration for performance. Apache processes .htaccess rules on every request, so for high-traffic sites, use server-level configuration (httpd.conf) instead of .htaccess (Apache, 2025).

For server-level Apache configuration:

<LocationMatch "/">
    SetEnvIf User-Agent "GPTBot" ai_crawler
    SetEnvIf User-Agent "ClaudeBot" ai_crawler
    SetEnvIf User-Agent "CCBot" ai_crawler
    Require not env ai_crawler
</LocationMatch>

This configuration uses environment variable matching with mod_authz_core to block requests. It provides better performance than mod_rewrite because access control evaluation happens earlier in Apache's request processing pipeline.

OpenLiteSpeed supports user-agent blocking through its access control rules in the web admin console or via configuration file directives. Caddy uses a more concise syntax in its Caddyfile:

@ai_crawlers {
    header User-Agent *GPTBot* *ClaudeBot* *CCBot*
}
respond @ai_crawlers 403

Both servers support the same pattern: match user-agent strings and return a blocking status code.

Implement server-level AI crawler blocking this week. Start with user-agent matching for the top eight AI crawlers and add IP-based blocking using DNS TXT records. Test your configuration with curl using custom user-agent strings to verify blocking works correctly. Monitor your access logs for any legitimate traffic that may be caught by your blocking rules.

Citations: Nginx (2025) Nginx Documentation; Cloudflare (2025) AI Crawler Management Guide; Apache (2025) Apache HTTP Server Documentation.