Robots.txt Syntax: 10 Best Practices That Work in 2026

The robots.txt file must live at the root of the origin. Googlebot looks for /robots.txt before any other path. A file at /subdirectory/robots.txt is ignored. Google's 2025 robots.txt documentation confirms this requirement for all crawler protocols. Use HTTPS for the origin. Googlebot respects only the HTTPS version.

Each record must specify a user-agent. The wildcard User-agent: * applies to all crawlers except those matched by a more specific rule. Google's crawler identifies as Googlebot. Image crawlers use Googlebot-Image. Video crawlers use Googlebot-Video. Storing a rule under User-agent: Googlebot does not affect Googlebot-Image unless you include it explicitly. The 2025 robots.txt specification by Google clarifies this inheritance model.

Disallow: / blocks every crawler from every page. This directive tells Googlebot the entire site is off limits. A 2025 analysis from Search Engine Land reported that 8 percent of the top 10,000 sites accidentally used a blanket disallow during site migrations. The result was a complete drop from search results for weeks. Use Disallow: with no value to allow crawling of all paths.

The Allow directive overrides a Disallow for a specific path. Place allow rules before disallow rules in each record group. Google's 2025 robots.txt testing tool validates override behavior. A common pattern: Disallow: /api/ then Allow: /api/public/. This allows the public API paths while blocking internal endpoints.

Googlebot stops reading after 500 KB of robots.txt data. Everything past that limit is ignored. The 2025 Google developers documentation on robots.txt sets this hard cutoff at 500 KB for each file. Large sites with autogenerated rules often exceed this limit without knowing it. Each rule consumes space. Trim unused directives.

Add a Sitemap: directive at the end of the file. Use the absolute URL. Google supports multiple sitemap entries. Example: Sitemap: https://example.com/sitemap.xml. Google's 2025 sitemap documentation shows this directive helps Googlebot discover sitemaps faster than manual submission alone. Do not mix sitemap directives inside user-agent groups. Place them at the file footer.

Google provides a robots.txt testing tool inside Search Console. It simulates Googlebot behavior against any rule set. A 2025 Ahrefs survey found that 34 percent of SEO professionals never tested robots.txt changes before deploying them. Untested rules cause blocked pages. Use the live testing endpoint to verify each URL pattern.

Crawl-delay is not supported by Google. It works for Bing, Yandex, and smaller crawlers. Use Crawl-Delay: 10 under a specific user-agent record for non-Google bots. Google relies on server response time and the setting in Search Console to manage crawl rate. The 2025 Bing Webmaster Tools documentation confirms crawl-delay support. Do not use crawl-delay in the Googlebot record.

Block staging environments, internal search result pages, infinite calendar archives, and parameter-based duplicates. Google's 2025 crawling advice recommends blocking paths that produce no unique content value. Each blocked path frees crawl budget for productive pages. Example: Disallow: /search/ and Disallow: /tag/ for a blog.

After updating robots.txt, watch the Crawl Stats report in Search Console for at least two weeks. A sharp drop in total crawl requests may indicate an overblocking disallow rule. A 2026 case study from Moz showed a site that lost 60 percent of crawls after a misconfigured dynamic robots.txt. Monitor crawl frequency per day. Blocking too many paths reduces how many pages Google indexes.

You verify the robots.txt file is at the root HTTPS origin. You check file size against the 500 KB limit. You review each user-agent group for overlapping or contradictory rules. You test ten high-value page paths using the Search Console robots.txt tester. You confirm the sitemap directive is present and points to a valid sitemap URL. You check Search Console crawl stats for anomalies. You review the staging subdomain to ensure it has a separate disallow-all rule. You identify unused patterns that waste file space.

Note the gap. This post synthesizes 2025 and 2026 data from 4 sources: Google Search Central robots.txt documentation, Search Engine Land 2025 robots.txt audit report, Ahrefs SEO survey 2025, and Moz 2026 technical SEO case study. 3 non-public enterprise migration audits uncovered the disallow-all pattern frequency. Replication required.

Audit quarterly.