Creating a list of all the pages on a website may seem like a simple task. In practice, no single method is sufficient to ensure complete coverage. Between pages excluded from internal linking, content generated by JavaScript, and URLs ignored by search engines, a comprehensive inventory remains a challenging goal to achieve without cross-referencing multiple data sources.
Orphan pages and JavaScript rendering: the blind spots of a site inventory
Most guides on the subject recommend checking the XML sitemap or launching a crawl of internal links. Both approaches cover a large portion of URLs, but they share a flaw: they only find what is linked or declared.
Related reading : Discover all the essential trends and analyses in the business world
Orphan pages (those that exist on the server without being linked from any other page) escape any crawl based on navigation. They may represent old content, test pages, or forgotten landing pages after a campaign.
The other blind spot concerns modern sites that load their content via JavaScript. A classic crawler sends an HTTP request and reads the raw HTML returned by the server. If links or pages only appear after executing the script on the browser side, the simple crawl does not detect them at all. Recent recommendations emphasize the need to add a browser rendering step to the crawling process to capture these invisible URLs.
You may also like : The latest fashion trends to discover to refresh your style this season
A directory like the one that lists all the pages of the Voiloo site illustrates an approach where the sitemap is publicly exposed and updated centrally, reducing the risk of missing pages for visitors.
XML Sitemap and Google Indexing: why the official list remains incomplete
The sitemap.xml file is often presented as the reference source for knowing all the URLs of a site. Its role is to signal to search engines the pages that the owner wants to see indexed. It does not reflect the entirety of what is online.
Several situations create a gap between the sitemap and reality:
- Pages may be present on the server but absent from the sitemap, either due to forgetting to update it or by deliberate choice (noindex pages, internal resources).
- Google may decide not to index certain URLs declared in the sitemap if it considers them duplicated, of low value, or technically problematic.
- Dynamic URLs (search filters, sorting parameters, paginated versions) may exist without ever appearing in the sitemap or search results.
The sitemap remains a useful starting point, but it does not constitute proof of completeness. It indicates what the webmaster declares, not what the site actually contains.
The site: operator in Google and its limitations
Typing “site:mondomaine.fr” in Google displays the indexed pages. This method provides a quick overview, but the displayed number fluctuates from one query to another and does not correspond to the actual total of indexed pages. Google does not guarantee the display of the entirety of its index for a given domain.
Recently published pages may take several days to appear. Deindexed or filtered pages no longer appear, even if they are still accessible via their direct URL. The site: operator measures visibility in Google, not the existence of pages.
Building a comprehensive page list: the layered approach
The most reliable method for creating a complete inventory relies on cross-referencing multiple sources, each compensating for the shortcomings of the others. This logic of successive layers is the only way to approach a comprehensive result.
The first layer involves retrieving the robots.txt file and the declared sitemaps. The robots.txt can reveal entire directories that the site owner wishes to hide from bots, but which do indeed exist.
The second layer is the crawl of internal links. A tool crawls each page accessible from the homepage by following all the links. This step identifies the actual navigation structure of the site and all the interconnected pages.
The third layer adds JavaScript rendering. Instead of only reading the static HTML, the crawler executes the JavaScript code of each page to discover links and content loaded dynamically. This step has become essential on sites built with frameworks like React, Vue, or Angular.
The fourth layer targets orphan pages. To find them, one must compare the list obtained from the crawl with other sources: server logs, Google Search Console, old versions of the site via the Wayback Machine, or exports from the CMS.
Deduplication and stability of the list
On large sites, the crawl often generates thousands of URLs pointing to the same content with different parameters. Feedback emphasizes the need to deduplicate URLs and limit the crawl rate to obtain a stable and usable list.
Without deduplication, the same product sheet can appear under dozens of variants (sorted by price, by reviews, by color). The final list inflates artificially and loses all utility for an audit or migration.
Case of showcase sites and small sites: a simpler inventory but not without pitfalls
On a showcase site of a few dozen pages, the problem seems less acute. The sitemap generally covers all content, and a quick crawl is enough to browse everything. However, pitfalls still exist.
Pages created by extensions (contact forms, thank you pages, automatically generated privacy policy pages) are frequently forgotten in manual inventories. Broken redirects or old URLs still accessible can also skew the count.
A showcase site owner who wants to verify their inventory saves time by cross-referencing the sitemap with a crawl, even a basic one. The comparison between the two lists highlights discrepancies in just a few minutes.
No method alone guarantees a perfect list. Cross-referencing sources remains the only approach that withstands technical blind spots, regardless of the size of the site.