How to Run an Indexation Audit and Close Coverage Gaps

An indexation audit answers one question precisely: of the pages you want in the index, how many are actually there, and for every page that is not, why not? The method is to build a definitive list of the URLs you want indexed, compare it against what search engines have actually indexed, and then categorise every discrepancy by cause. Indexation is where technical SEO turns into traffic, so this audit often surfaces the highest-impact issues on a site.

## Define your canonical set of indexable URLs

Before you can find gaps, you need a clean reference list of URLs that should be indexed. This is harder than it sounds, because most sites do not actually know their own URL inventory. Assemble it from:

- XML sitemaps

- A full crawl of the site following internal links

- CMS or database exports of published content

- Server logs showing URLs that receive traffic

Reconcile these sources. URLs that appear in some but not others are early warning signs: content with no internal links, sitemap entries for pages that no longer exist, or pages that exist but are never linked.

## Compare intended versus actual indexation

With your reference set in hand, compare it against indexation reporting. You are looking for two kinds of mismatch:

1. **Wanted but not indexed.** Pages you care about that search engines have excluded.

2. **Indexed but not wanted.** Pages in the index that should not be there, such as staging URLs, parameter variants, internal search results or thin pages.

Both matter. The first is lost opportunity; the second dilutes your site's perceived quality and wastes crawl attention.

## Categorise every exclusion by cause

The heart of the audit is explaining each exclusion. Most fall into recognisable buckets:

- **Blocked by robots.txt.** The page cannot be crawled, so it usually will not be indexed properly.

- **Marked noindex.** A directive, sometimes left in place by accident after a launch or migration.

- **Canonicalised to another URL.** The page points elsewhere as the preferred version.

- **Duplicate without a clear canonical.** Search engines chose a different URL as canonical than you intended.

- **Crawled but not indexed.** Often a quality or value signal; the page was seen but judged not worth indexing.

- **Discovered but not crawled.** Frequently a crawl budget or internal linking issue.

- **Soft 404.** A page returning 200 that looks empty or error-like.

Each cause implies a different remedy, so resist the urge to apply a single blanket fix.

## Pay special attention to two failure modes

Two categories deserve extra scrutiny because they are common and frequently misdiagnosed.

**Crawled but not indexed** usually points to content quality, duplication or thinness rather than a technical block. Improving or consolidating the content, strengthening internal links, and removing near-duplicates tends to help more than any technical toggle.

**Conflicting signals** are the silent killer of indexation. A page that is in the sitemap but also noindexed, or canonicalised to a URL that is itself noindexed, or disallowed in robots.txt while carrying a canonical, sends contradictory instructions. Audit specifically for these combinations, because tools often report only the symptom, not the conflict.

## Check the directives are internally consistent

For a sample of representative URLs across each template, verify that all indexation signals agree:

- The HTTP status is 200.

- The page is not blocked in robots.txt.

- There is no unintended noindex.

- The canonical points to itself (or deliberately elsewhere).

- The URL appears in the sitemap only if it is the canonical, indexable version.

- Internal links point to this canonical URL, not to redirects or alternates.

When these align, indexation problems usually resolve themselves over time. When they conflict, no amount of resubmission will help.

## Prioritise by value, not by count

Not all coverage gaps are equal. A thousand excluded parameter URLs you never wanted indexed are a tidiness issue; a handful of excluded high-intent commercial pages are a revenue issue. Rank findings by the business value of the affected pages so effort goes where it pays.

Doing this reconciliation across large sites repeatedly is exactly the kind of work that benefits from purpose-built tooling, which is part of why neart.ai builds enterprise-grade products in this space.

## Practical takeaway

Build a clean reference list of the URLs you actually want indexed, compare it against what is indexed, and explain every discrepancy by cause. Watch especially for conflicting directives and for pages that are crawled but not indexed, fix internal consistency before resubmitting anything, and prioritise the gaps by the value of the pages involved.