Meta AI Crawler: Understanding Meta's Primary AI Training Bot

Meta AI crawler, operating under the user-agent Meta-ExternalAgent , is Meta's dedicated bot for collecting training data for its generative AI models, including the LLaMA series and Meta AI assistant features. As Meta accelerates its AI development, understanding this crawler's behavior has...

Distinct Identity and Purpose

Meta-ExternalAgent is separate from FacebookBot (facebookexternalhit) and serves exclusively for AI training data collection. This separation was formalized in 2024 as Meta expanded its AI model training infrastructure. The crawler does not handle social sharing features, link previews, or any Meta platform functionality. Its sole purpose is gathering training data for AI model development (Meta, 2025).

Request Characteristics

Meta-ExternalAgent sends requests with consistent HTTP headers including a reference URL to Meta's crawler documentation. The crawler supports gzip compression, HTTP/2, and respects Cache-Control and ETag headers. Its IP ranges are published via Meta's _meta DNS TXT record and are distinct from FacebookBot's IP ranges, enabling precise identification in access logs (Cloudflare, 2025).

Crawl Prioritization

Meta's AI crawler prioritizes content based on several factors. It favors English-language content with high information density, technical documentation, news articles, and academic-style writing. The crawler uses machine learning to evaluate content quality during the crawl process, deprioritizing thin or low-value pages. This selective approach means that not all pages on a site receive equal crawl attention (Search Engine Journal, 2025).

Volume and Frequency

Meta-ExternalAgent exhibits moderate crawl volumes compared to search engine crawlers, typically 2 to 6 requests per second during active sessions. However, it maintains persistent crawl sessions over days or weeks rather than crawling intermittently. This steady-state behavior makes it easier to predict and manage compared to bursty crawlers like FacebookBot.

Rate Limiting and Management

Standard rate limiting techniques work effectively with Meta-ExternalAgent. Returning HTTP 429 triggers exponential backoff with increasing intervals. Operators can also use CDN-level rules to cap request rates from Meta's published IP ranges. Because the crawler is well-behaved and documented, server-level rate limiting is usually sufficient without requiring full blocking.

Audit your access logs for Meta-ExternalAgent traffic and compare it against FacebookBot volumes. Implement separate rate limiting rules for each user-agent if you have distinct policies for social features and AI training. Document Meta-ExternalAgent in your AI crawler inventory for ongoing traffic pattern analysis.

Citations: Meta (2025) Meta-ExternalAgent Documentation; Cloudflare (2025) AI Crawler Management Guide; Search Engine Journal (2025) Managing AI Crawlers with Robots.txt.