How to structure 5,000 pages on a static site

Five thousand pages is not an absurd number. A news site reaches it in a few years. A catalog, in weeks. The problem is not generating the files, but keeping the structure coherent once nobody can review page by page anymore.

This tutorial organizes the work into seven steps. The examples use Astro, a framework that generates static sites, but the principles apply to any generator. The search rules come from the Google Search Central documentation, consulted on October 8, 2026.

A static site is one in which every page is generated as a ready-made HTML file during the build, the stage that turns code and content into the final files. The server only delivers those files, without assembling anything on each visit.

Step 1: design the URL hierarchy before the content

The URL is the most expensive part to change later. Every change requires a permanent redirect and updating every internal link that points to the old address. So decide the pattern before publishing the first page.

A common pattern on content sites has two or three levels: section, sometimes subsection, and the final page. For example, /guides/finance/how-to-build-a-budget/.

Google’s URL structure guide gives the practical rules:

• Readable words instead of long ID numbers.
• Hyphens to separate words, not underscores.
• Few parameters: session, sorting and tracking codes multiply URLs without changing the content.
• One case: Google treats /APPLE and /apple as different URLs. Standardize everything in lowercase.

Also decide on the trailing slash. /page/ and /page are distinct URLs. Pick one form and always use it, including in internal links.

Step 2: turn content into data with rules

With 5,000 pages, the content needs a model: the list of fields each page type has (title, description, date, category, author) and the rules for each field.

In Astro, this is done with content collections. The collections documentation shows how to declare a schema, the formal description of the fields, so that the build fails when a file arrives without a date or with a nonexistent category. The same page states that collections generated at build time can be cached between builds and are suitable for “tens of thousands” of entries. In other words, 5,000 pages are within the expected range.

Two decisions help a lot:

• Separate identity from text. The page identifier (the slug, the final part of the URL) and the category live in their own fields, not derived from the title. That way, fixing a title does not change the URL.
• Use closed lists. Categories and topics come from a list defined in code. A typo becomes a build error, not a new category with a single page.

The technical details of collections are in Astro Content Collections.

Step 3: generate every route from a single list

On a static site, every URL must be known at build time. In Astro, a dynamic page exports the getStaticPaths() function. The routing reference explains that each returned item has params, which become the URL and only accept text, and optional props, which carry data to the page without appearing in the URL.

This tutorial’s recommendation is to have a single place that builds the list of every route on the site. The pages, the sitemap and the checks all come from that place. If each part of the system calculates URLs on its own, sooner or later the sitemap will list a page that does not exist, or a link will point to an old address.

This arrangement can be verified in Token & Tech’s own code. The file src/lib/routes.ts builds the route list, and a single page file, src/pages/[...path].astro, uses that list in getStaticPaths(). The name [...path] is a “rest” parameter, which accepts several URL segments at once. The sitemap is generated from the same list, and the code throws an error if two routes have the same path.

Step 4: create list pages only when they have content

Category, topic and author pages are the ones that multiply the most. With 5,000 articles and dozens of topics, it is easy to generate hundreds of list pages, many with one or two items.

A few simple rules prevent empty pages:

• Do not generate the page for a topic with no published items.
• Mark as noindex utility pages, such as internal search results and thank-you pages. The noindex tag asks the search engine not to include the page in its index.
• Do not put in the sitemap anything marked noindex.

In Token & Tech’s code, the route list skips topics without published articles and marks the search and thank-you pages as non-indexable. The sitemap only includes indexable routes.

Step 5: paginate long lists the right way

A category with 800 articles does not fit on one page. Astro’s paginate() function, described in the routing reference, splits the list into numbered pages. The default size is 10 items per page, adjustable with the pageSize option. Each page receives the previous and next links in page.url.prev and page.url.next.

Google’s pagination guide adds the search rules:

• Each page in the sequence has its own URL.
• Each page points its canonical to itself, not to the first one. The canonical is the tag that indicates which URL is the main version of a piece of content.
• Pages link to each other in sequence with ordinary <a href> links.
• Do not use # to number pages: Google ignores whatever comes after the #.
• Google no longer uses rel="next" and rel="prev", although other search engines may.

Filtered or re-sorted variations of the same list should stay out of the index, with noindex or a robots.txt block, according to the same guide.

Step 6: build the sitemap and the canonicals

The sitemap is the XML file that lists the URLs you want indexed. The sitemaps.org protocol and Google’s guide set the limit: 50,000 URLs or 50 MB uncompressed per file. Beyond that, it is split into several sitemaps, gathered by a sitemap index. With 5,000 pages, one file is enough.

Three details from Google’s guide, updated in July 2026:

• Google ignores <priority> and <changefreq>.
• <lastmod> is only used if it is consistent and verifiable. It should change when the main content changes, not when the year in the footer changes.
• All URLs in a sitemap must be from the same host, according to the protocol.

If you do not want to write your own generator, you can use the @astrojs/sitemap integration. It creates a sitemap-index.xml, splits files every 45,000 entries by default (the entryLimit option) and lets you exclude pages with filter().

For the canonical, Google’s documentation on duplicate URLs recommends a self-referencing canonical, with an absolute URL, and consistency between canonical, sitemap, redirects and internal links. On a static site, this follows naturally from Step 3: if every part uses the same route list, the URL is the same everywhere.

Step 7: link the pages and check everything at build time

Google’s links guide sums up the rule: every page that matters should receive a link from at least one other page on the site. Links must be <a> elements with an href and have descriptive text, not “click here”.

Across 5,000 pages, the link structure usually has three layers:

• Navigation: the section menu and the breadcrumb trail, which shows where the page sits in the hierarchy.
• Lists: paginated categories and topics, which ensure every article is reachable.
• Context: links within the text to related content.

Then automate the checking. A post-build check can open every URL in the sitemap and verify that the file exists, that it has a title, a description and a canonical equal to its own URL, and that each internal link leads to a generated file. Token & Tech does this with the scripts/check-dist.mjs script. The topic is covered in detail in How to test internal links and routes automatically, and search within a site of this size in How to implement internal search on an Astro site.

What about crawl budget?

Crawl budget is the number of pages Google’s crawler is willing to visit on a site in a given period. It worries a lot of people, but Google’s own guide, updated in July 2026, says who it is meant for: sites with around 1 million pages that change weekly, sites with 10,000 or more pages that change daily, or sites with many URLs in the “Discovered – currently not indexed” status in Search Console. Google calls these figures a rough estimate.

A static site with 5,000 pages that rarely change is below these ranges. In the editorial team’s assessment, effort pays off more in useful content, stable URLs and well-made links than in crawl tuning.

Key takeaways

• Decide URLs and the content model first. They are the most expensive parts to change with thousands of pages live.
• Build every route in one place and make it the source of the pages, the sitemap and the canonicals.
• Do not generate empty lists, paginate long ones with their own URL and canonical, and keep utility pages out of the index.
• At 5,000 pages, one sitemap is enough and crawl budget is rarely the bottleneck. Automate checks at build time so structural errors never go live.