How to automate technical SEO audits

A template change removes the meta description from every category page. Another marks an entire section as noindex by mistake. Neither breaks the site, and both can go unnoticed for weeks.

A technical SEO audit is the check of the elements that allow a search engine to find, read and index your pages: titles, descriptions, canonical, indexing directives, sitemap, links, performance. Done by hand, it becomes a report that quickly goes out of date. Automated, it becomes a test that runs with every release.

This tutorial organizes automation into four layers, from the cheapest to the slowest, and ends with the case of Token & Tech itself.

The four layers

Not every check needs to run at the same time. In the editorial team’s assessment, a practical split is this:

1. Static rules over the build. Objective checks on the generated HTML: there is a title, there is a description, the canonical is correct, there is no noindex where there should not be. They run in seconds and should block the release.
2. Lab audit. Lighthouse loads pages in a controlled browser and measures performance, accessibility, best practices and SEO. It runs in minutes, on a sample of pages.
3. Field data. Metrics from real users, which only exist after publication and change over weeks.
4. Index status. What Google has actually indexed, queried through the Search Console API.

Layers 1 and 2 go into the release pipeline. Layers 3 and 4 are monitoring: they run on a schedule and generate alerts, not blocks.

Layer 1: a script over the generated site

Start with what is objective and cheap. For a static site, the build produces a folder of HTML. A script reads the sitemap, opens the file for each URL and checks a list of rules. If it finds an error, it exits with a non-zero exit code, and the release process stops.

Rules that usually apply to any site:

• Title present, not empty and not duplicated within the same language.
• Meta description present and not empty. The Lighthouse documentation itself warns that its audit only checks this and “does not evaluate the quality” of the text. A length limit, if you want one, is your own rule.
• Canonical equal to the URL published in the sitemap.
• No sitemap page with noindex. Google accepts the directive both in the <meta name="robots"> tag and in the X-Robots-Tag HTTP header, and it only works if the page is not blocked in robots.txt, because the crawler needs to access the page to read the rule (Google Search Central). A page that is both in the sitemap and noindex is almost always a configuration error.
• A single H1 per page, if that is your site’s convention.
• Internal links that point to existing files. The step-by-step for this part is in How to test internal links and routes automatically.

A minimal skeleton, just as an example, in Node.js with no dependencies:

// example: check-seo.mjs: runs after the build
import { readFileSync, existsSync } from "node:fs";

const SITE = "https://example.com";
const errors = [];
const sitemap = readFileSync("dist/sitemap.xml", "utf8");
const urls = [...sitemap.matchAll(/<loc>([^<]+)<\/loc>/g)].map((m) => m[1]);

for (const url of urls) {
  const file = `dist${url.slice(SITE.length)}index.html`;
  if (!existsSync(file)) { errors.push(`no file: ${url}`); continue; }
  const html = readFileSync(file, "utf8");
  if (!/<title>[^<]{10,}<\/title>/.test(html)) errors.push(`${url}: title`);
  if (!/<meta name="description" content="[^"]+"/.test(html)) errors.push(`${url}: description`);
  if (!html.includes(`<link rel="canonical" href="${url}"`)) errors.push(`${url}: canonical`);
  if (/<meta name="robots" content="[^"]*noindex/.test(html)) errors.push(`${url}: noindex in sitemap`);
}

if (errors.length) { console.error(errors.join("\n")); process.exit(1); }
console.log(`OK: ${urls.length} URLs`);

The regular expressions above depend on the exact format of the HTML your generator produces. For less predictable HTML, an HTML parser is safer.

Two observations. First, the script only sees the HTML: an X-Robots-Tag configured on the server does not show up here, and needs to be checked with requests to the published site. Second, separate errors from warnings. An error blocks; a warning, such as a description that is too long, shows up in the log and does not prevent the release.

Layer 2: Lighthouse CI

Lighthouse is an open-source automated tool from Google that audits pages for performance, accessibility, SEO and other categories. It can run in Chrome DevTools, on the command line, as a Node.js module or through PageSpeed Insights (overview).

Lighthouse CI is the version made for continuous integration. According to the getting started guide, the lhci autorun command collects results, applies assertions and uploads the reports. For static sites, the staticDistDir option points to the build folder, where the HTML to be audited lives. For sites with their own server, the startServerCommand and url options specify how to start the server and which pages to test.

A minimal configuration, just as an example:

// example: lighthouserc.js
module.exports = {
  ci: {
    collect: { staticDistDir: "./dist", numberOfRuns: 3 },
    assert: {
      assertions: {
        "categories:seo": ["error", { minScore: 0.9 }],
        "categories:performance": ["warn", { minScore: 0.8 }],
      },
    },
    upload: { target: "filesystem" },
  },
};

Points from the configuration documentation that change the outcome:

• Assertion level. warn prints the failure but does not change the exit code; error makes the process exit with a non-zero code. That difference is what decides whether the failure blocks the release.
• Number of runs. The default is 3 runs per URL, because lab measurements fluctuate. The default aggregation method is optimistic, which uses the value “most likely to pass” across the runs. If you want a stricter bar, there are median and pessimistic.
• Report destination. The temporary-public-storage option is the simplest, but the documentation recommends it only if you accept that the reports will be accessible to anyone with the link. For private projects, prefer filesystem or your own server.

A tip from the guide itself: on sites with many problems, turn on assertions gradually, treating the failures as a to-do list. A test that always fails ends up being switched off.

And an important limit. Lighthouse’s SEO score adds up presence audits: does the page block indexing? Does it have a description? Each SEO audit carries the same weight in the score, according to the documentation. A score of 100 means the technical requirements checked are present. It does not mean the page will rank.

Layer 3: field data, on a schedule

Lighthouse measures in the lab: a defined device, network and location, so that the result is reproducible. Field data comes from real users, usually from the Chrome User Experience Report (CrUX), and forms a distribution. web.dev recommends using field data to prioritize and lab data to debug and test changes before publishing.

The three Core Web Vitals and their “good” thresholds, measured at the 75th percentile of visits, are (web.dev):

• LCP (Largest Contentful Paint, time until the largest visible element loads): up to 2.5 seconds.
• INP (Interaction to Next Paint, response time to interactions): up to 200 milliseconds.
• CLS (Cumulative Layout Shift, how much the layout shifts): up to 0.1.

Lighthouse does not measure INP, because in the lab there is no real interaction. It uses Total Blocking Time (TBT) as a rough proxy. A page can have good TBT and poor INP.

To query these numbers automatically, the PageSpeed Insights API returns the Lighthouse result and, when available, the field metrics. An API key is optional, but recommended for frequent automated queries. The same page states that Google plans to stop including CrUX data in this API and points to the CrUX API instead. If you are starting now, it is worth building the field query on that API from the outset.

Because field data changes slowly, running this layer with every release makes no sense. A weekly query, saved to a file or spreadsheet, shows the trend.

Layer 4: what Google has indexed

The last layer answers the question the others do not: has Google indexed the page, and with which canonical? The urlInspection.index.inspect method of the Search Console API returns the status of a URL in the version that is in the index. The documentation itself warns that, for now, only that status is available: you cannot test the indexability of the live URL through the API.

The call requires OAuth authentication with access to the property in Search Console and the webmasters or webmasters.readonly scope. Use the read-only one. Store the credential in your automation’s secrets vault, never in the repository.

A reasonable use: once a week, inspect a sample of important URLs and recently published ones, and alert when the canonical chosen by Google differs from the declared one.

Where each layer runs

Put the layers together like this:

• With every build, before publishing: layer 1 (blocking) and, if time allows, layer 2 on a sample of template pages, such as the home page, an article and a category.
• On a schedule: layers 3 and 4, with alerts.
• Live, right after publishing: a few requests to the real site to check HTTP status, redirects and headers that the build does not show.

How to connect all this to a safe automated release is the subject of How to set up automated deployment securely.

Documented case: the Token & Tech audit

The site you are reading uses layer 1 as a release gate. The scripts/check-dist.mjs script, called by npm run check, runs over the folder generated by the build and uses no external dependencies.

For each sitemap URL, it checks that the file exists and that the page has an <html lang> matching the language, a title with a minimum length and no duplicate within the same language, a description, a canonical equal to the URL, exactly one H1 and an existing sharing image. A sitemap page with noindex is an error. It then cross-checks hreflang: each version must reference itself, have x-default and receive the link back.

The script also checks that robots.txt points to the sitemap, that required files exist and that configuration and data files have not ended up in the published folder. A title over 70 characters and a description over 165 generate a warning, not an error.

If there is any error, the script lists them all and exits with code 1. The publishing script only proceeds if the build and this check pass. The site does not use Lighthouse CI in its pipeline; layers 2 to 4 are outside the automated process described here.

Takeaways

• Split the audit into layers: static rules and lab tests at release time, field data and index status on a schedule.
• Build layer 1 first. It is cheap, objective and catches the most costly errors, such as a wrong noindex or canonical.
• In Lighthouse CI, only error blocks. Use warn for what is still being fixed, and do not upload reports from a private project to public storage.
• Treat the Lighthouse score as a requirements check, not a ranking forecast.