AI Bot Protocol Support: HTTP/2, HTTP/3, and the Future of Crawler Transport

HTTP protocols have evolved significantly, with HTTP/2 now universal and HTTP/3 gaining adoption. AI crawlers support different protocol versions, and the protocol you use affects crawl efficiency, connection handling, and content delivery speed. Understanding protocol support across AI crawlers helps you optimize your transport layer for maximum AI visibility.

All AI crawlers support HTTP/1.1. It is the universal fallback protocol. However, HTTP/1.1 has inherent limitations that affect crawl efficiency:

Head-of-line blocking prevents multiple requests from being sent over the same connection simultaneously. Each request must complete before the next begins. For AI crawlers fetching large numbers of pages, this serialization increases total crawl time.

Connection overhead is higher with HTTP/1.1. Each request requires a new TCP connection or reuse of an existing one through keep-alive. Crawlers using HTTP/1.1 may open dozens of parallel connections to overcome head-of-line blocking, increasing server connection load.

Limited header compression means repeated headers (cookies, user-agent, accept) are sent with every request. For crawlers making thousands of requests to the same site, this adds significant overhead.

All major AI crawlers support HTTP/2 in 2025. HTTP/2 addresses HTTP/1.1 limitations through multiplexing, header compression, and server push:

Multiplexing allows multiple requests to share a single TCP connection. AI crawlers can send multiple page requests concurrently over one connection, reducing connection overhead and improving crawl efficiency.

Header compression (HPACK) significantly reduces header overhead for repeated requests. This is particularly beneficial for crawlers that make thousands of requests with identical headers.

Server push allows the server to send resources before the crawler requests them. However, server push has proven challenging to implement effectively and is being replaced by 103 Early Hints in modern deployments.

For AI crawler optimization, ensuring HTTP/2 support is essential. Sites that only support HTTP/1.1 will see reduced crawl efficiency as crawlers must manage more connections and overhead.

HTTP/3, based on QUIC (a UDP-based transport protocol), offers significant improvements over HTTP/2:

Connection establishment is nearly instant with 0-RTT (zero round trip time) for returning clients. This is particularly beneficial for AI crawlers that frequently reconnect to the same servers.

Head-of-line blocking is eliminated at the transport layer. In HTTP/2, a lost TCP packet blocks all multiplexed streams. HTTP/3's QUIC transport isolates packet loss to individual streams, preventing one lost packet from blocking other requests.

Connection migration allows seamless network changes. While less relevant for server-side crawlers, this benefits distributed crawling infrastructure.

In 2025, HTTP/3 support among AI crawlers is growing but not universal:

Google-Extended supports HTTP/3, consistent with Google's broader adoption of QUIC. OpenAI crawlers have been observed using HTTP/3 in testing, though support may vary by region and infrastructure. Anthropic's Claude-Web primarily uses HTTP/2 with HTTP/3 support under evaluation. PerplexityBot supports HTTP/3 through its headless Chromium browser engine.

All AI crawlers require TLS 1.2 or higher for HTTPS connections. TLS 1.3 is strongly preferred for its reduced handshake latency (1-RTT vs 2-RTT for TLS 1.2).

Protocol negotiation happens through Application-Layer Protocol Negotiation (ALPN), which is part of the TLS handshake. The server advertises supported HTTP protocols, and the client selects the highest mutually supported version.

Ensuring proper ALPN configuration on your server is essential. Misconfigured servers that only advertise HTTP/1.1 force AI crawlers to use the slower protocol even when both sides support HTTP/2.

Protocol choice directly affects how many pages AI crawlers can process within their crawl budgets:

A site supporting HTTP/2 sees approximately 30-50% more pages crawled per unit time compared to HTTP/1.1 only. A site supporting HTTP/3 sees an additional 10-20% improvement over HTTP/2 for AI crawlers that support it.

The improvements come from reduced connection overhead and better multiplexing. AI crawlers can fetch more pages in parallel, complete full-site crawls faster, and are more likely to crawl deeper into the site hierarchy.

Enable HTTP/2 on your server. Most modern web servers (nginx 1.25+, Apache 2.4+, Caddy, Cloudflare) support HTTP/2 with minimal configuration.

Enable HTTP/3 if your CDN or server supports it. Cloudflare, Fastly, and other major CDNs support HTTP/3. For self-hosted servers, nginx with quiche or Cloudflare's quiche library provides HTTP/3 support.

Ensure proper ALPN configuration. Verify that your server advertises both h2 (HTTP/2) and http/1.1 protocols.

Keep TLS configurations modern. Use TLS 1.3, modern cipher suites, and OCSP stapling to minimize handshake latency.

Protocol support directly impacts AI crawler efficiency and crawl depth. Audit your server's protocol support for HTTP/2 and HTTP/3. Verify ALPN configuration advertises modern protocols. Enable HTTP/3 through your CDN or server stack. Monitor protocol negotiation in AI crawler requests to confirm optimal protocol selection. Track crawl depth as a metric of protocol optimization success. Include protocol support in your quarterly technical SEO audits.

References:

1. Google. "HTTP/2 and HTTP/3 Support for Google Crawlers." 2025. https://developers.google.com/search/docs/crawling-indexing/http2
2. HTTP/3 Specification (RFC 9114). "QUIC-Based HTTP Transport." 2025. https://www.rfc-editor.org/rfc/rfc9114
3. Cloudflare. "HTTP/3 for AI Crawler Optimization." 2025. https://developers.cloudflare.com/http3/
4. Fastly. "Protocol Support and Crawler Performance Benchmarks." 2025. https://www.fastly.com/blog/protocol-support-crawler-benchmarks