A large website should not treat 20,000 URLs as if they all carry the same commercial value.
The practical indexing challenge is to make sure search engines can discover the right URLs efficiently while low-value, duplicate and obsolete pages do not dominate the estate.
Create URL priority tiers
A useful model is:
Tier 1: revenue-critical
Core service, product, category, location and conversion pages.
Tier 2: authority-building
High-quality guides, case studies and supporting content that feed commercial journeys.
Google Search Essentials separates technical eligibility from whether Google chooses to crawl, index and serve a URL. Discovery work can remove barriers and improve signalling, but it should never be presented as a guarantee of indexation or rankings.
Tier 3: utility
Useful but low-commercial-value pages such as help content.
Tier 4: low-value or excluded
Duplicates, account pages, filters, internal searches and other URLs that should not compete for indexing attention.
Monitor by tier
If a Tier 1 page disappears from the index, that is urgent. If an obsolete filter URL disappears, that may be desirable.
This is why a simple “indexed/not indexed” count is not enough. You need context.
Sitemaps can reflect structure
Large estates often benefit from segmented sitemaps by content type or section. This makes it easier to identify where problems are concentrated.
For example, a product sitemap suddenly losing 30% of URLs is more actionable than a single combined file changing by an unexplained amount.
Internal linking distributes importance
Search engines learn from how the site links internally. Important categories and services should receive prominent, relevant links. Deep pages with no inbound path are harder to discover and easier to neglect.
Continuous monitoring is an operations problem
At scale, manual checks stop working. You need repeatable monitoring and exception reporting.
IndexFlow is designed around that model: keep the discovery layer visible and surface issues that deserve human attention.
What success looks like
For an operational next step, daily indexing and sitemap monitoring helps turn those checks into a repeatable process rather than a one-off inspection.
Do not aim for “100% of all URLs indexed.” That can be the wrong target.
Aim for:
- high coverage of commercially important canonical URLs;
- fast discovery of meaningful new pages;
- low contamination from errors and obsolete URLs;
- clear visibility when indexation patterns change.
Key takeaways
- Prioritise large estates by commercial value.
- Segment sitemaps where it improves diagnosis.
- Measure indexing quality, not raw index size.
- Use automated monitoring to find exceptions rather than manually checking thousands of URLs.
Apply the idea without creating more noise
Start with the page's actual job and the reader's next decision. Check the evidence already available, identify the gaps preventing that decision, and improve those before adding another page or campaign. A useful change should make the journey clearer, not simply make the estate larger.
Measure the result against a baseline. For informational content that may mean qualified impressions, clicks and progression to a relevant commercial page; for commercial content it should extend to enquiries, sales or another meaningful conversion. This keeps optimisation tied to business value rather than publishing volume.
Implementation and measurement
Treat the recommendation as a decision process rather than a publishing task. Define the page or campaign's intended audience, the problem it should solve and the next useful action before changing anything. Compare the current experience with that job, then prioritise the smallest changes that remove genuine friction or ambiguity.
Use evidence at the point it matters. Primary guidance can establish factual constraints; first-party data, examples and experience can explain what those constraints mean in practice. Internal links should expand a concept or move a qualified reader to the next relevant step, not exist simply to increase link counts. The anchor should make the destination understandable before the click.
After implementation, record a baseline and review the metric that matches the job. Discovery work should improve qualified visibility; educational content should create useful progression into deeper or commercial pages; conversion work should be judged on enquiries, sales or another meaningful outcome. Avoid claiming causation from a single short observation window.
This also creates a maintenance loop. When products, evidence, search behaviour or official guidance changes, refresh the affected passage and its linked destinations rather than producing another overlapping URL. That keeps the estate coherent and reduces the risk of thin, stale or competing pages.
Questions to ask before the next change
Use four checks. Relevance: does the page still answer the query or problem it targets? Evidence: are important factual claims supported and current? Journey: can a reader reach the next useful explanation, proof point or commercial destination without hunting through navigation? Outcome: is there a metric that tells you whether the page is helping the business rather than merely existing?
These checks are deliberately simple. They prevent optimisation from becoming a list of disconnected SEO tasks and make future refreshes easier to prioritise. If a page cannot pass them, improve the weak part before creating another URL that covers substantially the same ground.