Index Audit: Aligning Search Console Data with Crawl Reality

Indexation is not binary. A page can be crawled but not indexed, indexed but not ranking, or ranking with a canonical that points to a different page entirely. The gap between what a site publishes and what Google actually stores in its index is often wider than teams realize. An index audit measures that gap and provides a roadmap for closing it.

Google's index is not a mirror of the web. It is a curated subset, filtered through quality signals, deduplication logic, and canonicalization rules. Understanding those rules is the prerequisite for any index audit.

Begin in Google Search Console. Navigate to the Index Coverage report and export the full list of indexed and excluded URLs. Classify each entry into its root cause category. The most common categories are:

• Crawled but not indexed. Google found the page but chose not to include it. This typically means the page was deemed low quality, thin, or duplicative. Pages in this state may benefit from content expansion, internal link strengthening, or removal from the sitemap.
• Discovered but not crawled. Google knows about the URL (from sitemaps or links) but has not yet allocated crawl budget. This is common for new pages on large sites. Increase internal linking and ensure the page is prominent in the sitemap.
• Excluded by noindex. The page carries a <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex HTTP header. Verify this was intentional. A common mistake is a noindex tag left on a staging page that was accidentally published.
• Excluded by canonical mismatch. The page has a canonical tag pointing to a different URL, and Google respects it. If this was unintentional, correct the canonical.
• Alternative page with proper canonical. Google selected a different canonical URL than the one you specified. Investigate why your canonical signal was overridden.

A 2025 study by Merkle analyzing 1,000 domains found that 28 percent of pages submitted in XML sitemaps were in the "Crawled but not indexed" state, with the highest rates on ecommerce product pages and blog archives [1].

Your XML sitemap is the primary mechanism for telling Google which URLs you consider important. An inaccurate sitemap dilutes that signal. Audit your sitemap against these criteria:

• Every URL in the sitemap returns a 200 (OK) HTTP status code.
• Every URL is indexable (no noindex, no blocked-by-robots, no canonical to a different page).
• No URL in the sitemap triggers a redirect chain longer than one hop.
• The sitemap contains no more than 50,000 URLs per file (or 50 MB uncompressed) as specified by Google's sitemap protocol [2].
• The <lastmod> dates in the sitemap are accurate. Google uses lastmod as a freshness signal. Incorrect dates can cause Google to recrawl unchanged pages or skip genuinely updated pages.

Use your crawler to cross-validate each sitemap URL against its live response. Any URL that fails one of the criteria above should be removed from the sitemap until the issue is resolved.

Run a canonicalization audit across the entire indexed URL set. For each page, compare three values: the page's self-referencing canonical tag, the page's internal link anchor text target, and the URL Google indexed (from Search Console's URL Inspection tool). Discrepancies between these three values create confusion about which URL should rank.

The most common canonical problems in 2026 are pagination-related. Ecommerce sites frequently leave canonical tags pointing to page 1 of a category from page 2 and beyond, when the paginated pages themselves are not indexed as separate entities. The correct pattern is for paginated pages to canonicalize to themselves, with a rel="prev" and rel="next" markup or, equivalently, a single-page view with infinite scroll.

Index bloat occurs when Google indexes a large number of low-value or duplicate URLs, diluting the site's overall index quality signal. The typical sources are parameter-based URLs (sort, filter, search), printer-friendly versions, and paginated archives.

Identify index bloat by running a Search Console query for "Google-selected canonical" entries that differ from your self-declared canonical. A high volume of Google-selected canonicals indicates that the search engine is trying to deduplicate pages that your site structure should not be generating in the first place.

Run the full index audit monthly as part of your ongoing SEO monitoring. The four-step process (coverage classification, sitemap audit, canonical check, bloat assessment) takes approximately two hours for a site of 50,000 pages once the initial data pipelines are set up. Track the ratio of "Indexed" to "Submitted in sitemap" as your primary indexation health metric. A healthy site maintains a ratio above 0.85. If your ratio falls below 0.70, prioritize resolving the exclusion causes identified in Step 1 before any other optimization work.

---

References

1. Merkle. (2025). "SEO Indexation Study: Sitemap Coverage Analysis Across 1,000 Domains." Merkle Digital Research.
2. Google. (2025). "Sitemaps Best Practices." Google Search Central Documentation.
3. Metha, A. (2025). "Canonicalization in the Age of AI Overviews." Search Engine Journal, April 2025.