Technical SEO Audit for the AI Era: A Step-by-Step Guide (2026)

person

AI SEO Intelligence

calendar_today October 11, 2026

schedule 12 min read

Technical SEO Audit for the AI Era: A Step-by-Step Guide (2026)

A technical SEO audit answers one question: can search engines - and now AI search systems - fetch, render, understand and index the pages you care about? The checks below follow that order. Every rule in this guide comes from Google's or Bing's current documentation, linked as you go, because technical SEO advice ages fast and the official pages are where changes show up first.

Key Takeaways

robots.txt controls crawling, not indexing. Google says it is not a mechanism for keeping a page out of search; use noindex for that.
Canonicals and lastmod are hints Google weighs, not commands. Redirects are the strongest consolidation signal; Google ignores sitemap priority and changefreq.
Core Web Vitals thresholds: LCP within 2.5 seconds, INP of 200 ms or less, CLS of 0.1 or less, at the 75th percentile. INP replaced FID on March 12, 2024.
Google renders JavaScript; most AI crawlers do not. Server-side or static rendering covers both.
There is an AI layer now: Google's Search generative AI control in Search Console, and Bing directives that limit how Copilot uses your content.

Step 1: Crawl Access

Start with robots.txt, because nothing else matters if crawlers cannot fetch the page. Google's robots.txt documentation sets the rules worth checking:

Size: Google processes the first 500 KiB of the file and ignores the rest. The RFC 9309 standard requires parsers to handle at least that much.
Errors: a 4xx response (except 429) is treated as if no robots.txt exists - everything is crawlable. A 5xx response makes Google stop crawling the site for 12 hours, then fall back to a cached copy for up to 30 days. A robots.txt that returns 500 is a site-wide outage for crawling.
Purpose: robots.txt is not access control and does not keep pages out of the index. Google's introduction is explicit: block indexing with noindex or password protection instead.

Then check AI crawlers separately: the search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) decide whether you can be cited, and the training bots (GPTBot, ClaudeBot, CCBot) are a separate policy decision. Check your CDN or firewall too; a bot-protection rule can block a crawler that robots.txt allows. Our Robots.txt and AI Bot Access Rules checks cover both.

Step 2: Indexing

In Search Console's page indexing report, look at why pages are excluded, not just how many. Two problems deserve special attention:

Accidental noindex. A noindex left over from staging, or sent as an X-Robots-Tag header, removes pages silently.
Soft 404s. Google defines a soft 404 as a page that tells the user it does not exist but returns a 200 status code. Its crawl-error guidance: return 404 or 410 if the content is gone, 301 if it moved. Our Soft 404 check tests this.

Since August 2026, also open Settings > Search generative AI in Search Console. The control includes sites in AI Overviews and AI Mode by default; if someone switched it to exclude, the site disappears from both.

Step 3: XML Sitemaps

Google's sitemap documentation gives the limits: 50 MB uncompressed or 50,000 URLs per file, with a sitemap index above that. It also says Google uses lastmod only when it is consistently and verifiably accurate, and ignores priority and changefreq entirely. So the audit questions are simple: does the sitemap list only canonical, indexable URLs, and does lastmod change only when the content does? Make sure robots.txt points to it.

For Bing, its webmaster guidelines recommend IndexNow to notify it when URLs are added, updated or removed. IndexNow submissions are shared with all participating engines, which per the IndexNow FAQ include Bing, Naver, Seznam, Yandex and Yep.

Step 4: Canonicals and Duplicates

Google calls a canonical preference a hint, not a rule, and may pick a different canonical than you did. Its consolidation guide ranks the signals: redirects and rel="canonical" are strong signals, sitemap inclusion is a weak one. In the audit, check that every indexable page has a self-referencing canonical, that canonicals point to 200-status indexable URLs, and that signals agree - the canonical, internal links and the sitemap should all name the same URL. Our Self-Referencing Canonical check flags mismatches.

Step 5: Redirects

From Google's HTTP status documentation: 301 and 308 are strong signals that the target should be canonical; 302 and 307 are weak signals. Google's crawlers follow up to 10 redirect hops. Long chains waste crawl effort and risk breaking before the destination, so collapse every chain into a single hop and update internal links to point straight at the final URL. The Redirect Chains check finds them.

Step 6: JavaScript Rendering

Google renders pages with an evergreen version of Chromium: pages go through crawling, rendering and indexing, and all 200-status pages enter the rendering queue. Google still calls server-side rendering or pre-rendering a great idea, and describes dynamic rendering as a workaround rather than a recommended solution.

AI crawlers change the calculation. In Vercel and MERJ's crawl analysis, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and others fetched pages without executing JavaScript. Content that only exists after client-side rendering may be fine for Google and invisible to AI search. Compare the raw HTML (View Source, or curl) with the rendered page; our JavaScript Content Gap check does this automatically. More detail in which AI crawlers render JavaScript.

Step 7: Core Web Vitals and Performance

The current thresholds from web.dev, measured at the 75th percentile of page loads:

Metric	Measures	Good
LCP (Largest Contentful Paint)	Loading	2.5 s or less
INP (Interaction to Next Paint)	Responsiveness	200 ms or less
CLS (Cumulative Layout Shift)	Visual stability	0.1 or less

INP replaced FID as a Core Web Vital on March 12, 2024. Google's page experience documentation says Core Web Vitals are used by its ranking systems, while other page experience aspects do not directly help a site rank higher. Use field data from Search Console's Core Web Vitals report to find failing templates, then PageSpeed Insights to diagnose individual URLs.

Step 8: Mobile and HTTPS

Google completed the move to mobile-first indexing in October 2023, so the mobile version of a page is the one that gets indexed. Check that it has the same content, links, structured data and metadata as desktop.

HTTPS should be in place everywhere, with no mixed content. Google announced HTTPS as a lightweight ranking signal in 2014; today's page experience documentation lists secure serving as a self-check rather than a ranking factor. Treat it as a baseline, not a lever.

Step 9: Structured Data

Google's structured data policies require markup to be a true representation of the page content, and forbid marking up content readers cannot see. Bing's guidelines say the same. Audit that required properties are present and match the visible page. Two 2026 changes to account for: FAQ rich results stopped appearing in Google Search on May 7, 2026, and Google's generative AI guide says structured data is not required for its AI features. Keep markup accurate for the rich results that still exist; do not expect it to make a page visible in AI answers on its own.

Step 10: The AI-Search Layer

Both search engines now document how their AI features use your pages, and an audit should check those controls are set on purpose:

Google: a page must be indexed and eligible for a snippet to appear in AI features. nosnippet, data-nosnippet, max-snippet and noindex limit what can be shown. The Search generative AI control in Search Console must be on include.
Bing and Copilot: Bing's guidelines apply to Bing search, Copilot and grounding results alike. NOARCHIVE prevents content from being used in Copilot responses, NOCACHE limits Copilot to the URL, title and snippet, and NOSNIPPET may limit Copilot citation quality.
Training: Google-Extended (a robots.txt token, not a separate crawler) and the training bots of other vendors control model training, separately from search visibility.

A directive added years ago for an unrelated reason - a site-wide noarchive, say - can now quietly remove you from AI answers.

How to Prioritize the Fixes

Fix in the order the steps run: an issue early in the chain blocks everything after it. A robots.txt that returns 500, a stray noindex or a firewall that blocks crawlers outranks every performance or markup issue. Within each step, fix templates before individual URLs. Our SEO audit report template shows how to turn findings into a prioritized plan, and the best SEO audit tools covers which tool to use for each step.