FacebookBot: Meta's Web Crawler for Social Features and AI

FacebookBot identifies with multiple user-agent strings depending on its purpose. The primary user-agent is facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php). A secondary user-agent Meta-ExternalAgent handles AI-specific crawling. Both originate from IP ranges published in Meta's DNS records. The user-agent differentiation allows operators to distinguish between social feature requests and AI training data collection (Meta, 2025).

FacebookBot drives link preview generation, Open Graph tag parsing, and article summary extraction for Facebook, Messenger, and Instagram. These requests are essential for proper social sharing behavior. Separately, Meta uses FacebookBot for collecting training data for its AI models, including LLaMA and other generative AI systems. Blocking FacebookBot entirely would break link previews while also preventing AI training (Cloudflare, 2025).

FacebookBot generates substantial request volumes, particularly when URLs are shared across Meta's platforms. Each share triggers a fresh crawl to update the preview. During viral content events, FacebookBot traffic can spike dramatically, sometimes exceeding 100 requests per hour from multiple IPs. The AI training component operates at more moderate, sustained rates focused on content discovery rather than share-triggered updates (Jetpack, 2025).

FacebookBot respects standard robots.txt directives. For granular control, operators can use:

User-agent: facebookexternalhit
Disallow: /path/to/no-share/

User-agent: Meta-ExternalAgent
Disallow: /

Meta also supports the nositelinkssearchbox meta tag for controlling how content appears in Meta's AI features. The separate user-agent tokens give operators flexibility to allow social sharing while blocking AI training data collection.

Because FacebookBot requests can spike unpredictably based on sharing activity, operators should implement caching strategies for link preview endpoints and Open Graph metadata. Static OG tags with proper cache headers significantly reduce the server load from FacebookBot's repeated share-triggered requests. This optimization also improves the social sharing experience for users by delivering faster preview rendering.

Review your access logs for both facebookexternalhit and Meta-ExternalAgent user agents. Consider implementing separate robots.txt rules for each token to maintain social sharing functionality while controlling AI training access. Optimize your Open Graph tag caching to reduce FacebookBot's server load impact.

Citations: Meta (2025) FacebookBot Documentation; Cloudflare (2025) AI Crawler Management Guide; Jetpack (2025) AI Crawler Traffic Analysis Report.