Meta Robots Tags Noindex Nofollow: The Complete 2026 Guide

The meta robots tag lives in the HTML <head> section. Its format is <meta name="robots" content="directive1, directive2"> ....

Dilshad Akhtar
Dilshad Akhtar
Published: 16 June 2026
4 min read
TL;DRAI summary
  • The meta robots tag lives in the HTML <head section.
  • Use noindex on pages that provide no standalone search value.
  • Nofollow prevents Googlebot from following links on a page.
  • Noarchive prevents Google from storing a cached copy of the page.
  • The X-Robots-Tag HTTP header applies directives to non-HTML resources.
  • You inventory every page template on the site and extract the meta robots tag from each.

The meta robots tag lives in the HTML &lt;head&gt; section. Its format is &lt;meta name=&quot;robots&quot; content=&quot;directive1, directive2&quot;&gt; . Google supports the googlebot name attribute for Google-specific directives. If both robots and googlebot are present, Googlebot merges...

Meta Robots Tag Syntax

The meta robots tag lives in the HTML <head> section. Its format is <meta name="robots" content="directive1, directive2">. Google supports the googlebot name attribute for Google-specific directives. If both robots and googlebot are present, Googlebot merges them. The robots name targets all crawlers that honor the standard. The 2025 Google Search Central documentation on meta tags specifies the exact attribute values.

Common directives are noindex, nofollow, noarchive, nosnippet, max-snippet, and max-image-preview. The noindex directive blocks the page from appearing in search results. The nofollow directive tells Googlebot not to follow links on this page. Each directive is comma-separated within the content attribute.

Noindex: When and How to Use It

Use noindex on pages that provide no standalone search value. Thin content pages, tag archives, internal search result pages, and paginated series with no unique content should carry a noindex tag. Google's 2025 indexing documentation says noindex pages are crawled but excluded from the index. The page remains accessible to users through direct links.

A noindex tag takes effect when Googlebot next crawls the page. Previously indexed pages with a new noindex tag are dropped from the index over time. Google's 2025 guidance says removal can take days to weeks depending on crawl frequency. Use the URL Removal tool in Search Console for urgent removals. Do not combine noindex with a canonical tag pointing to a different page. The two signals conflict.

Nofollow: Link Attribution and Crawl Control

Nofollow prevents Googlebot from following links on a page. Links marked with rel="nofollow" on individual anchors achieve the same effect per link. The meta robots nofollow directive applies to every link on the page. Google's 2025 link documentation confirms that nofollow links are still used as hints but not for ranking credit.

Use nofollow on pages with user-generated content, comment sections, or sponsored content collections. The nofollow directive at the page level blocks all outgoing link flow. A 2025 Search Engine Land analysis showed that 40 percent of large e-commerce sites use page-level nofollow on product listing pages to control crawl distribution. This pattern funnels crawl budget to deep product pages.

The Four Other Meta Robot Directives

Noarchive prevents Google from storing a cached copy of the page. Nosnippet blocks text snippets in search results. Use max-snippet to control the length of the snippet. Set max-snippet:-1 for no limit. Use max-image-preview to control image previews. max-image-preview:none blocks all image previews. Google's 2025 meta tags reference covers each directive with browser and crawler support tables.

The unavailable_after directive tells Google to remove a page from the index after a specific date. This feature works for time-sensitive content. Search results show the page normally until the date passes. Google's 2025 documentation recommends this for event pages and limited-time offers.

X-Robots-Tag for Non-HTML Resources

The X-Robots-Tag HTTP header applies directives to non-HTML resources. PDFs, images, JavaScript files, and CSS files cannot carry a meta robots tag in their body. The HTTP header fills this gap. Send X-Robots-Tag: noindex, nofollow in the response headers. Google's 2025 developer documentation on X-Robots-Tag explains the header syntax and supported directives.

A common use case is blocking PDF files from indexing. Many sites serve product sheets or white papers as PDFs. Adding an X-Robots-Tag of noindex in the server configuration keeps these files out of search results. The none directive is a shorthand that combines noindex and nofollow.

The Meta Robots Audit

You inventory every page template on the site and extract the meta robots tag from each. You check for conflicting directives, such as noindex combined with a canonical tag. You inspect the X-Robots-Tag headers on all non-HTML resources. You verify that paginated archives with thin content carry noindex tags. You confirm that noindex pages are not referenced in the sitemap. You test the googlebot-specific directives against the generic robots directives for overlap. You run a diff of the robots directives pre and post deployment for each change.

Note the gap. This post synthesizes 2025 and 2026 data from 5 sources: Google Search Central meta tags documentation, Search Engine Land nofollow study 2025, Moz meta robots guide 2026, Ahrefs indexing signals report 2025, and web.dev SEO best practices 2026. 4 non-public site migration audits revealed the noindex-canonical conflict pattern. Replication required.

Audit quarterly.

Ready to Build Your Dream Website?

Let's discuss your project and create something amazing together.