Sign in Sign up free

AI Crawlers List 2026: Every AI Crawler You're Accidentally Blocking

AI SEO Intelligence

May 25, 2026
Updated November 16, 2026
15 min read
AI Crawlers List 2026: Every AI Crawler You're Accidentally Blocking

Update, October 11, 2026: this post now carries a full AI crawlers list (31 user agents and tokens from 14 operators), checked against each operator's own documentation on October 11, 2026. We also corrected five things the May version got wrong: Anthropic's three bots were documented by August 2025, not first in February 2026; OpenAI does publish an IP list for OAI-AdsBot; Applebot-Extended does not crawl, so it has no traffic share; Perplexity's documentation never called Perplexity-User "an agent, not a bot"; and Google-Extended covers grounding as well as training.

Update, October 12, 2026: we wrote that ByteDance publishes no documentation for Bytespider. It does: a Chinese-language page on its Toutiao Search webmaster platform lists Bytespider's user agents, 10 IP ranges and a reverse-DNS check, and describes it as Toutiao's search crawler. The table and the verification list now say so; our Bytespider guide has the details.

Update, November 16, 2026: the "Search crawlers" item said that blocking them removes you from AI answers. That is stronger than two operators say: OpenAI writes that sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, "though can still appear as navigational links" (OpenAI), and Anthropic says blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results" (Anthropic). The rest of the post is unchanged.

The robots.txt file on most production sites was written before half the AI crawlers it now governs existed. Since 2024 the major AI companies have stopped using one user agent for everything: OpenAI, Anthropic, Perplexity, Meta, Amazon, Mistral and Google each run separate agents for model training, for their search index and for fetches a user asks for. Each of those agents reads robots.txt differently, and some do not read it at all.

Below is the current list, then what changed, how much these crawlers actually fetch, and a short self-audit for your own robots.txt.

Key Takeaways

  • There are now more than 30 AI user agents and robots.txt tokens worth knowing, from 14 operators. Most sites govern them through one User-agent: * rule written for something else.
  • Training and search are separate switches. Blocking GPTBot or ClaudeBot keeps your content out of training; it does not remove you from ChatGPT or Claude search. Those follow OAI-SearchBot and Claude-SearchBot.
  • User-triggered fetchers often ignore robots.txt by design. OpenAI says robots.txt "may not apply" to ChatGPT-User; Perplexity, Google, Meta and Amazon say much the same about their user-triggered fetchers. Anthropic and Mistral are the exceptions.
  • Some names are tokens, not crawlers. Google-Extended and Applebot-Extended never visit your site; they only tell Google and Apple how they may use content their main crawlers already fetched.
  • Training now dominates crawling. Cloudflare reported that 52% of crawler requests were for AI training in June 2026, up from 22% in spring 2025.

AI Crawlers List 2026

"Follows robots.txt" reports what each operator says in its own documentation, not what we measured.

Operator User agent / token What it does Follows robots.txt?
OpenAI GPTBot Crawls content that may be used to train OpenAI's models Yes
OpenAI OAI-SearchBot Builds the index behind ChatGPT search Yes
OpenAI ChatGPT-User Fetches a page when a ChatGPT user asks about it "may not apply"
OpenAI OAI-AdsBot Checks pages submitted as ChatGPT ads; not used for training Not stated
Anthropic ClaudeBot Collects content that may be used for training Yes
Anthropic Claude-SearchBot Indexes pages for Claude's search results Yes
Anthropic Claude-User Fetches a page when a Claude user asks about it Yes
Google Googlebot Google Search, including AI Overviews and AI Mode Yes
Google Google-Extended Token only: Gemini training and grounding; no crawler Token
Google Google-CloudVertexBot Crawls a site its owner asked to index for Vertex AI Agents Yes
Google Google-Agent Agents on Google infrastructure acting for a user Generally ignores; no token
Google Google-GeminiNotebook Fetches pages a user adds to a notebook (was Google-NotebookLM) Generally ignores
Apple Applebot Spotlight, Siri and Safari search; data may also train Apple models Yes; falls back to Googlebot rules
Apple Applebot-Extended Token only: opt-out from Apple model training; "does not crawl webpages" Token
Perplexity PerplexityBot Indexes pages for Perplexity search; not used for training Yes (recommended to allow)
Perplexity Perplexity-User Fetches a page when a Perplexity user asks about it "generally ignores"
Meta meta-externalagent Crawls for uses "such as training foundation AI models" Not stated
Meta meta-webindexer Lets Meta AI "cite and link to your content" Not stated
Meta meta-externalfetcher Fetches a page a user asked for "may bypass"
Meta meta-externalads Crawls for advertising use Not stated
Amazon Amazonbot Crawls for Amazon products; "may be used to train Amazon AI models" Yes
Amazon Amzn-SearchBot Search experiences such as Alexa; not used for training Yes; falls back to your rules for other search bots
Amazon Amzn-User Fetches information for a user "may not follow all"
Microsoft bingbot Bing Search and Copilot answers; no separate Copilot token Yes
Common Crawl CCBot Open web archive used to build many training datasets Yes
ByteDance Bytespider Crawler for Toutiao Search, per ByteDance's webmaster page (in Chinese); reported to collect AI training data Not stated
DuckDuckGo DuckAssistBot Fetches pages for DuckDuckGo's AI answers; not used for training Yes (takes effect after 72 hours)
Mistral MistralAI-Training Collects data to train Mistral models Yes
Mistral MistralAI-Index Search index; "not used for generative AI training of any kind" Yes
Mistral MistralAI-User Fetches a page for a user of Mistral's assistant Yes
You.com YouBot Search index for You.com Yes ("fully respects")

Verifying a crawler

User-agent strings are easy to fake, so check the source IP against the operator's published list: OpenAI (gptbot.json, searchbot.json, adsbot.json, chatgpt-user.json), Anthropic (bots.json, one list for all three bots), Google (common-crawlers.json and a separate list for user-triggered agents), Apple (applebot.json), Perplexity (perplexitybot.json, perplexity-user.json), Amazon (IP lists), Microsoft (bingbot.json), Common Crawl (ccbot.json), DuckDuckGo (duckassistbot.json), Mistral (index, user), ByteDance (10 ranges plus a reverse-DNS check on its webmaster page) and You.com (68.67.112.0/24). Meta does not publish one.

Fallback rules

Two rows deserve a second look. Apple says that if your robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. Amazon says the same about Amzn-SearchBot and the rules you give other search bots. A rule written for Google can therefore decide Apple's and Amazon's access too.

Four Kinds of AI Crawler

The list falls into four groups, and the cost of blocking each one is different:

  • Training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent, MistralAI-Training, Amazonbot, and Bytespider, which ByteDance documents as a search crawler but which is reported to collect training data) decide whether your future content is used to train models. Blocking them is a content-policy choice with no direct effect on AI search visibility.
  • Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, MistralAI-Index, YouBot, Amzn-SearchBot, and Googlebot and Bingbot) build the indexes that AI answers are drawn from. Blocking them removes you from those answers. Google's AI Overviews and AI Mode use Googlebot; there is no separate AI Overviews crawler to allow or block.
  • User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, Amzn-User, MistralAI-User, Google-Agent) fetch a page because a person asked the assistant about it. Most of their operators say robots.txt may not apply, because a user, not an automated crawl, made the request. Only Anthropic and Mistral say these fetchers honor it.
  • Control tokens (Google-Extended, Applebot-Extended) have no crawler. Google-Extended tells Google whether Gemini may train on, and ground answers in, content Googlebot already fetched; Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". Applebot-Extended is Apple's training opt-out.

Most robots.txt files we audit do not distinguish among these groups. A Disallow: / under User-agent: * meant to stop training also blocks every search crawler that has no section of its own, while leaving most user-triggered fetchers unaffected.

What Changed in 2025-2026

Anthropic: three bots and a published IP list

Anthropic's crawler documentation describes three bots - ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user-triggered) - and was already listing all three by August 2025. In early 2025 the page described ClaudeBot alone and said Anthropic did not publish IP ranges. It now links a bots.json list and states that Anthropic's bots honor robots.txt and the non-standard Crawl-delay directive. Anthropic also notes that blocking by IP "may not work correctly or persistently guarantee an opt-out", so robots.txt is the reliable switch.

OpenAI: an ads bot and a narrower robots.txt promise

OpenAI's bot documentation now lists four agents. OAI-AdsBot, added in 2026, only visits pages submitted as ChatGPT ads, its data "is not used to train" models, and its IPs are published in adsbot.json. The page also says robots.txt rules "may not apply" to ChatGPT-User because a user starts those requests, that each bot's setting is independent of the others, and that a robots.txt change takes about 24 hours to reach ChatGPT search.

Google: user-triggered agents

Google's list of user-triggered fetchers now includes Google-Agent, used by agents running on Google's infrastructure to act on a user's request. Like Google's other user-triggered fetchers, it "generally ignore[s] robots.txt rules", and it has no robots.txt token. Google-NotebookLM was renamed Google-GeminiNotebook.

Meta, Amazon and Mistral split their crawlers

Meta now separates meta-externalagent (training and other uses), meta-webindexer (which "helps us cite and link to your content in Meta AI's responses"), meta-externalfetcher (user-triggered, "may bypass robots.txt rules") and meta-externalads. FacebookBot is no longer on the page. Amazon runs Amazonbot, Amzn-SearchBot and Amzn-User. Mistral runs MistralAI-Training, MistralAI-Index and MistralAI-User. If you want to appear in Meta AI or Alexa answers while opting out of training, these are new decisions.

Perplexity: the user fetcher ignores robots.txt

Perplexity's crawler documentation recommends allowing PerplexityBot, which it says is not used for model training, and states that Perplexity-User, because a user requested the fetch, "generally ignores robots.txt rules". In August 2025 Cloudflare reported that Perplexity also crawled with a generic Chrome-on-macOS user agent from IPs outside its published ranges, and removed it from its verified-bot list. In the same test, ChatGPT-User fetched robots.txt and stopped when it was disallowed. If you need Perplexity-User off your site, the published perplexity-user.json range and a firewall rule are the tool, not robots.txt.

Your CDN may be blocking AI crawlers for you

Robots.txt is no longer the only place access is decided. In September 2026 Cloudflare changed its defaults for newly added ad-supported domains to disallow AI training and to block AI agents on pages with ads, and its block settings now also apply to Googlebot, Bingbot and Applebot. A site can have a perfect robots.txt and still turn crawlers away at the CDN.

How Much These Crawlers Actually Fetch

Cloudflare publishes the most consistent numbers, from traffic across its network:

  • Training dominates. 52% of crawler requests were for AI training in June 2026, up from 22% in spring 2025 (Cloudflare, July 2026).
  • Crawling far outruns referrals. In July 2025 Anthropic crawled about 38,000 pages for every visitor it referred, and OpenAI about 1,100 (Cloudflare, August 2025). Across 2025, Anthropic's ratio peaked at around 500,000 to 1 (Cloudflare 2025 Year in Review). Cloudflare notes Claude's apps send no referrer, which inflates Anthropic's ratio.
  • GPTBot grew fastest. Its share of AI and search crawling rose from 4.7% to 11.7% between July 2024 and July 2025; ClaudeBot's went from 6% to nearly 10%.
  • The most blocked. GPTBot, ClaudeBot and CCBot were the user agents most often fully disallowed in robots.txt files in 2025.

How Discovery Mechanisms Are Shifting

Robots.txt is not the only file in this conversation. Since late 2024 a second one has spread: /llms.txt, a plain-text file at the root of your domain that lists the documents you want AI systems to read. Our companion post on llms.txt walks through the evidence: server-log studies show AI crawlers rarely request it, and Google's guidance says Google Search does not use such files. The short version: robots.txt decides whether a crawler can fetch you; llms.txt helps developer tools that read documentation, but it is not a lever for AI search visibility. Our audit reports it as information only.

There is a second layer most "let the AI bots in" guides skip past: even when a crawler is allowed and reaches your page, most of them never execute the JavaScript that would reveal your content. In Vercel and MERJ's December 2024 study, none of the major AI crawlers rendered JavaScript. Our companion post on JavaScript rendering across AI crawlers has the per-crawler matrix. Allowing GPTBot or OAI-SearchBot through robots.txt is necessary but not sufficient if your content only exists after a client-side render.

For bot-by-bot detail, see our guides to GPTBot, ClaudeBot and PerplexityBot.

A 10-Minute Self-Audit for Your robots.txt

Open your site's robots.txt (append /robots.txt to your domain) and work through this:

  1. Find your wildcard group. Look for User-agent: * followed by Disallow: /. Every crawler without its own group inherits it - and Applebot and Amzn-SearchBot may inherit your Googlebot or search-bot rules instead.
  2. Decide on training separately from search. A common 2026 policy blocks the training crawlers and tokens and allows everything else. Several User-agent lines can share one set of rules:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
Disallow: /

User-agent: *
Allow: /
  1. Check that search crawlers are not caught by accident. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and, if you care about them, meta-webindexer, Amzn-SearchBot and MistralAI-Index should not fall under a blanket Disallow: /.
  2. Treat user-triggered fetchers as a firewall question. A robots.txt rule stops Claude-User and MistralAI-User, but ChatGPT-User, Perplexity-User, meta-externalfetcher, Amzn-User and Google-Agent may fetch anyway. If you must keep them out, use the operator's published IP list at the firewall.
  3. Check your CDN settings. If you use Cloudflare or a similar service, confirm its AI crawler and training settings match what your robots.txt says.
  4. Verify against the operators' own pages. OpenAI, Anthropic, Google common crawlers and user-triggered fetchers, Apple, Perplexity, Meta, Amazon and Mistral. These pages changed several times in the past year.
  5. Run an audit. Our AI bot access check reads your robots.txt for 15 search and AI user agents, one scored result each, and flags blocked search crawlers and bots with no rule. A further informational result shows what your file says to 13 more agents from Meta, Amazon, Mistral, You.com, DuckDuckGo and Google, without affecting the score.

What This Means for Your Roadmap

Make the training-vs-search decision explicit. If you do not have separate groups for training crawlers and search crawlers, you are coupling two independent decisions. Writing them out takes ten minutes and documents the choice for whoever edits the file next.

Decide on the newer search agents. meta-webindexer, Amzn-SearchBot, MistralAI-Index and YouBot feed answers in Meta AI, Alexa, Mistral and You.com. If you allow everything by default you already have them; if you block by default, they are probably blocked without anyone deciding.

Re-check twice a year. OpenAI, Anthropic and Google all changed their crawler documentation in the past year, and Meta, Amazon and Mistral now document agents that most robots.txt files never mention.

Update - shipped 2026-05-25. Five crawlers joined our default audit catalog: Claude-SearchBot, Claude-User, Applebot-Extended, Perplexity-User and OAI-AdsBot. October 11, 2026: Amazonbot, meta-externalagent and MistralAI-User, held back in May, are now part of a separate informational result alongside ten more agents; it shows their access without changing scores.

The robots.txt file is one of the cheapest configuration assets a site owns and one of the most neglected. "Implicit consent through silence" worked when there were three crawlers to think about. With more than thirty, the silence is doing more work than most site owners intend.

Want to see which AI crawlers your robots.txt allows, blocks or never mentions? Run a free audit at hybridranking.com.

Sources

  1. Overview of OpenAI Crawlers - OpenAI
  2. Does Anthropic crawl data from the web, and how can site owners block the crawler? - Anthropic
  3. Google's common crawlers - Google Search Central
  4. Google's user-triggered fetchers - Google Search Central
  5. About Applebot - Apple
  6. Perplexity Crawlers - Perplexity
  7. Meta Web Crawlers - Meta for Developers
  8. Amazonbot - Amazon Developer
  9. Mistral crawlers - Mistral AI
  10. CCBot - Common Crawl
  11. DuckAssistBot - DuckDuckGo
  12. YouBot - You.com
  13. Perplexity is using stealth, undeclared crawlers - Cloudflare, August 2025
  14. The crawl-to-click gap - Cloudflare, August 2025
  15. Cloudflare Radar 2025 Year in Review
  16. Content Independence Day, one year on - Cloudflare, July 2026
  17. Accountable mixed-use AI crawlers - Cloudflare, September 2026
  18. The rise of the AI crawler - Vercel and MERJ, December 2024
  19. 关于Bytespider (About Bytespider) - Toutiao Search Webmaster Academy, ByteDance
The HybridRanking Advantage

See What AI Search Sees on Your Page

Run a free audit: AI crawler access, structured data, content structure and technical SEO, with a fix for every issue.