Sign in Sign up free

Most AI Crawlers Still Don't Render JavaScript in 2026 - And It's Not Even Close

AI SEO Intelligence

June 14, 2026
Updated October 13, 2026
8 min read
Most AI Crawlers Still Don't Render JavaScript in 2026 - And It's Not Even Close

Update, October 13, 2026: the section "How Our Audit Treats This" described the JavaScript Content Gap check as catching content that "only appears after client-side execution". The check does not render the page. It flags a page when it finds framework markers (such as __NEXT_DATA__ or a #root element with almost no text) and the raw HTML has fewer than 200 words (a warning) or fewer than 50 (a failure). A page with normal text in the HTML and one JavaScript-injected section is not caught. The rest of the post is unchanged.

A recurring claim has been making the rounds in 2025-2026 AI-SEO writeups: the major AI crawlers have caught up to Googlebot, they render JavaScript now, and your single-page app is fine. On the evidence available in Q2 2026, it is wrong almost everywhere it matters. Every AI-native crawler that has been measured - OpenAI's three, Anthropic's ClaudeBot, PerplexityBot - fetched pages without executing JavaScript in Vercel and MERJ's crawl analysis, and none of the vendors documents rendering at all. The only confirmed renderers in the AI-relevant set (Googlebot, Applebot) are pre-existing search-engine crawlers reusing infrastructure built years before generative AI existed.

Key Takeaways

  • Vercel and MERJ's crawl analysis (December 2024) found that GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Bytespider and Meta-ExternalAgent do not execute JavaScript, and that CCBot does not render pages. None of these vendors documents rendering behavior either way. Googlebot and Applebot are the only AI-relevant crawlers that ship a full renderer. Bingbot's rendering is limited and Microsoft itself recommends server-rendered HTML. A November 2025 SEJ analysis of AI crawlers found that AI crawlers cannot execute most JavaScript and cannot see dynamically loaded content - client-side rendered content is invisible to them. Vercel's data shows the ChatGPT and Claude crawlers do fetch JavaScript files (11.5% and 23.8% of their requests) but never execute them.
  • Onely's February 2026 analysis found that 42% of JavaScript-rendered content never gets indexed by AI systems - the largest quantified estimate of the visibility gap published to date.
  • Of the user-triggered fetchers, ChatGPT-User was measured as non-rendering; Claude-User and Perplexity-User have not been measured publicly. The "user-triggered means headless browser" assumption has no data behind it, and the likely reason is per-fetch latency cost inside chat responses.
  • Google-Extended is a robots.txt token, not a separate crawler. Google's crawler documentation says it has no user agent of its own; crawling is done by Google's existing crawlers, which render JavaScript. Disallowing it keeps your content out of Gemini training and grounding without affecting Google Search. Applebot-Extended functions as an opt-out signal for Apple Intelligence training, not as a separate crawler.
  • Traffic-share breakdowns by engine are not consistent enough to anchor a hard percentage, but the structural pattern is unambiguous: the dominant generative-AI products operate retrieval through non-rendering crawlers by deliberate architectural choice.

The Per-Crawler Rendering Map

Crawler Renders JS? Source
GPTBot No Vercel/MERJ crawl data, Dec 2024
OAI-SearchBot No Vercel/MERJ crawl data
ChatGPT-User No Vercel/MERJ crawl data
ClaudeBot No Vercel/MERJ crawl data
PerplexityBot No Vercel/MERJ crawl data
Bytespider No Vercel/MERJ crawl data
Meta-ExternalAgent No Vercel/MERJ crawl data
CCBot No Vercel/MERJ crawl data
Perplexity-User Not measured No public crawl data; vendor docs silent
Claude-User Not measured No public crawl data; vendor docs silent
Claude-SearchBot Not measured No public crawl data; vendor docs silent
OAI-AdsBot Not measured No public crawl data; vendor docs silent
Bingbot Partial / limited Microsoft docs
Googlebot Yes (headless Chromium) Google docs
Applebot Yes Apple docs
Google-Extended Control token (crawled by Googlebot, which renders) Google crawler docs
Applebot-Extended Opt-out token Apple docs

Eight crawlers measured as non-rendering, one partial, two that render (Googlebot and Applebot, both pre-existing search crawlers), four not measured, and two control tokens. We documented the full taxonomy - who launched what and when - in our companion piece on AI crawler blocking through Q3 2026.

The Googlebot-Gemini exception runs deeper than the table suggests. Martin Splitt confirmed in 2025 that Gemini AI uses the same Web Rendered Service (WRS) as Googlebot - the same headless Chromium infrastructure Google has maintained since 2015. Google-Extended is only the robots.txt switch that decides whether that crawled content may be used for Gemini; the crawling and rendering are Googlebot's. That is why Google's AI products can process JavaScript-heavy sites in ways that ChatGPT, Claude, and Perplexity cannot. This is a structural infrastructure advantage, not a crawl-time policy choice, and it is unlikely to close for the AI-native vendors in the near term.

Why User-Triggered Crawlers Don't Render Either

Here is the intuition most teams arrive at and then defend: "GPTBot is a training crawler, sure, it fetches static HTML. But ChatGPT-User fires when a real human asks for my page in real time. That has to be a headless browser, because the model is sitting there waiting." For OpenAI's ChatGPT-User the intuition is wrong: Vercel and MERJ measured it fetching pages without executing JavaScript. Claude-User and Perplexity-User have not been measured publicly and their vendors do not document rendering, but the structural argument below applies across all three.

The most likely reason is latency cost, not engineering capability. Every one of these vendors could ship a headless Chromium with their user-triggered crawler. What is harder to absorb is the per-page render wait inside the chat response: each rendered page adds non-trivial time on top of the model's own latency budget, and the user is sitting there waiting. No vendor has published a latency budget for user-triggered fetching, but the consistency of the architectural choice across OpenAI, Anthropic, and Perplexity points at a shared constraint of this shape. Google and Apple can renderer-fetch because their renderers are shared infrastructure paid for by organic search; the AI-native vendors had to choose, with a blank slate, and they all chose the same way.

What This Means for Single-Page Apps

If your primary content arrives via client-side JavaScript - the React/Vue/Svelte/Angular SPA shipped between 2018 and 2024, with content loaded by fetch() after the initial HTML response - the page that GPTBot, PerplexityBot, and ClaudeBot see is the empty shell. The <div id="root"> is empty. The headline, the article body, the product description, the schema markup - none of it is in the markup the crawler stores. The model, asked later to answer a question that would have referenced your content, has nothing in its index that matches.

The scale of the problem is not theoretical. Onely's February 2026 analysis put a number on it: 42% of JavaScript-rendered content never gets indexed by AI systems. That is not a rounding error or a fringe edge case - it is the majority-plus outcome for content that only exists in the rendered DOM. A 2025 Sitebulb survey of 295 SEO professionals found that 33.9% believe none of the LLM crawlers render JavaScript and another 29.8% had no idea - meaning the majority of practitioners have either correctly identified or are unaware of the gap, but few are treating it as a solved problem.

The remediations are the same ones the SEO community has been recommending since 2015 for Googlebot reliability, with the bar raised:

  • Server-side rendering (SSR) - Next.js, Nuxt, SvelteKit, Remix, etc., with primary content in the initial HTML response.
  • Static site generation (SSG) - pre-rendered HTML at build time. The same shape, more aggressive caching.
  • Hybrid / island architectures - server-rendered shell with client-side hydration, but with all citation-relevant content in the HTML.
  • Edge rendering / streaming SSR - pre-render at request time at the CDN edge.

What changes in 2026 is not which patterns work - those are unchanged - but how confidently you can skip them. "Googlebot renders JS pretty well, we can probably get away with CSR" was a defensible cost trade-off until 2024. The AI-cohort crawlers that now drive the bulk of AI referral traffic do not render. The trade-off no longer favors deferral.

AI Crawlers List: Which User-Agents to Allow in robots.txt

Rendering is only half of AI crawler access. A bot that robots.txt blocks never fetches your HTML at all, so moving to server-side rendering does nothing for it. These are the AI crawler user-agents that decide whether your pages can show up in ChatGPT, Claude, Perplexity and Google's AI features, grouped by what each one does:

Operator User-agent What it does
OpenAI GPTBot Crawls content that may be used to train OpenAI models
OpenAI OAI-SearchBot Builds the index behind ChatGPT search
OpenAI ChatGPT-User Fetches a page when a ChatGPT user asks for it
Anthropic ClaudeBot Crawls content for model training
Anthropic Claude-SearchBot Indexes content for Claude's search answers
Anthropic Claude-User Fetches a page when a Claude user asks for it
Perplexity PerplexityBot Builds the Perplexity search index (not used for training, per Perplexity)
Perplexity Perplexity-User Fetches a page when a Perplexity user asks for it
Google Google-Extended Controls whether content is used for Gemini training
Apple Applebot-Extended Controls whether content is used for Apple Intelligence training
Common Crawl CCBot Builds the open web dataset many models are trained on

Training and search are separate decisions. OpenAI's documentation states that disallowing GPTBot keeps content out of model training, while sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers." Blocking GPTBot alone does not remove you from ChatGPT search. A robots.txt that opts out of training but stays visible in AI search looks like this:

# Stay visible in AI search
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Opt out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

The user-triggered fetchers are left out on purpose. They act on a person's request, and two vendors say robots.txt may not stop them: OpenAI writes that for ChatGPT-User "robots.txt rules may not apply," and Perplexity says Perplexity-User "generally ignores robots.txt rules." Anthropic's help center, by contrast, documents opting Claude-User out through robots.txt.

Our AI Bot Access Rules check reads your robots.txt and reports which of these bots have explicit rules. The JavaScript Content Gap check covers the other half: whether the raw HTML those bots receive contains your content at all.

How Our Audit Treats This

Our audit applies one of its largest non-indexing-related penalties when the AI-visibility surface of a page depends on JavaScript-injected content - when the headline, primary copy, structured data, or main article body is absent from the raw HTML response and only appears after client-side execution. We rebalanced this weight in Q1 2026 specifically because the AI cohort is the traffic that matters now, and the AI cohort does not render. If your site's main content is JS-rendered, the JavaScript Content Gap check will surface it as such, with the SSR/SSG migration path as the recommendation. On interactive app pages, the Noscript Fallback check flags a missing <noscript> alternative for crawlers that cannot run JavaScript.

A 5-Minute Self-Check for Your Own Site

You do not need an audit tool to find out whether your site has the JS-only-content problem.

  1. Open the page (homepage, top blog post, top product page) in a browser.
  2. View Source (Ctrl/Cmd-U) - the raw HTML the server returned, exactly what a non-rendering AI crawler sees.
  3. Search the source for your headline, the first paragraph of body text, and any structured data (application/ld+json).
  4. If both are in the source, you are server-rendering (or pre-rendering at build). AI crawlers see your content.
  5. If View Source shows an empty <div id="root"> or a page shell with no actual copy, your site is client-side rendered and the non-rendering AI crawler cohort is currently storing an empty shell.

For a bypass-the-browser confirmation:

curl -A "GPTBot" https://your-domain.com/your-page | grep "your headline text"

If grep returns the line, your headline is in the raw response and AI crawlers see it. If grep returns nothing, your headline is being injected later by JavaScript.

What This Doesn't Mean

Five caveats, because the contrarian claim is narrower than it sounds.

  • JavaScript is not forbidden. Interactive widgets, analytics, sign-up flows, configurators, comments, chat - all of that runs on the client and should. The argument is specifically about the primary content the page exists to communicate: headline, article body, product description, structured data.
  • Googlebot has not stopped rendering. Google still renders. The point is that Google Search is no longer the only ranking surface: the crawlers behind ChatGPT search and Perplexity were measured not rendering, and Claude's search crawler has not been measured at all.
  • Dynamic content is not the problem. A news homepage that updates every five minutes is fine. The question is who renders it into HTML - your server (good for AI crawlers) or the user's browser (invisible).
  • Not every SPA page is broken. If your SPA pre-renders critical pages at build time, or if your hosting layer SSRs the cited routes, those pages are fine. The check is per-page, not per-framework.
  • "Not measured" means not measured, not yes. Perplexity-User, Claude-User, Claude-SearchBot and OAI-AdsBot have no public rendering data. Treating them as renderers is wishful; treating them as non-renderers is the conservative bet that matches every measured crawler from the same vendors.

Run a free audit at hybridranking.com on your homepage or a top article to see whether the shell-and-no-content pattern is showing up in your raw HTML. The JS-rendering signal is one of the 111 checks in our audit library - heuristic on SPA-framework markers plus raw-HTML word count, not a literal raw-vs-rendered DOM diff, so it covers the common case rather than every edge case. AI-assistant workflow: the same check catalogue is callable as native tools from Claude Desktop and Cursor - we shipped an MCP server so the assistant can pull our current weighting straight into your chat.

Sources

  1. OpenAI bot documentation
  2. Anthropic clarifies its three Claude bots - Search Engine Land, Feb 2026
  3. Perplexity Crawlers - official docs
  4. What AI Sees When It Visits Your Website - Search Engine Journal, Nov 2025
  5. 68 Million AI Crawler Visits Show What Drives AI Search Visibility - Search Engine Journal, Apr 2026
  6. Semantic SEO for AI Search: The Enterprise Guide to Citation-Driven Visibility - Onely, Feb 2026
  7. Gemini AI Renders JavaScript Like Googlebot: Martin Splitt - StanVentures, 2025
  8. JavaScript SEO Report 2025: Survey & Audit Data - Sitebulb
  9. Does Anthropic crawl data from the web, and how can site owners block the crawler? - Anthropic Help Center
  10. The rise of the AI crawler - Vercel and MERJ, Dec 2024
  11. Google's common crawlers (Google-Extended) - Google Search Central
The HybridRanking Advantage

See What AI Search Sees on Your Page

Run a free audit: AI crawler access, structured data, content structure and technical SEO, with a fix for every issue.