What Is ClaudeBot? Anthropic's Crawlers and How to Control Them
ClaudeBot is the web crawler Anthropic uses to collect content that may be used to train its Claude models. It is one of three bots Anthropic documents, and many robots.txt files still address it through older tokens Anthropic no longer lists. An August 2025 analysis by Paul Calvano found that anthropic-ai, a token absent from Anthropic's current documentation, was still among the three bots with the most Disallow rules on popular sites.
ClaudeBot also stands out in traffic data. In July 2025 Anthropic's crawlers fetched about 38,000 pages for every visitor Claude referred, according to Cloudflare - the highest ratio of any AI company Cloudflare tracked. This guide explains what ClaudeBot and its two siblings do, how to control each one, and how to tell real Anthropic traffic from imitations.
Key Takeaways
- ClaudeBot is a training crawler. Anthropic says it collects content that "could potentially contribute to their training".
- Anthropic runs three bots. Claude-SearchBot indexes pages for Claude's search results and Claude-User fetches pages when a Claude user asks about them.
- All three honor robots.txt. Anthropic states its bots respect "do not crawl" signals and also support the non-standard
Crawl-delaydirective. - Anthropic publishes an IP list at claude.com/crawling/bots.json, one list for all three bots; it did not publish IP ranges at all in early 2025.
- It reads raw HTML only. In Vercel's December 2024 measurements Claude's crawler downloaded JavaScript files but did not execute them.
What Is ClaudeBot?
ClaudeBot is Anthropic's crawler for model training. Anthropic's crawler documentation (updated April 7, 2026) says that if a site restricts ClaudeBot, its future materials "should be excluded from our AI model training datasets".
Anthropic does not publish full user-agent strings, only the tokens. Third-party bot directories show different strings and contact addresses for the same bot, so match on the token: ClaudeBot anywhere in the user agent.
Action: write rules as User-agent: ClaudeBot, and remove stale anthropic-ai or Claude-Web groups only after adding the current tokens.
ClaudeBot vs Claude-SearchBot vs Claude-User
Anthropic describes three robots. Each can be controlled separately:
| Bot | What it does | What blocking it does |
|---|---|---|
ClaudeBot |
Collects web content that may be used for training | Excludes your future content from training datasets |
Claude-SearchBot |
"navigates the web to improve search result quality for users" | Stops Anthropic indexing your content for search |
Claude-User |
Fetches a page when a Claude user asks a question about it | "may reduce your site's visibility for user-directed web search" |
Anthropic's documentation already listed all three by August 2025; in January 2025 it still described ClaudeBot alone. Unlike OpenAI, which says robots.txt "may not apply" to its user-triggered ChatGPT-User, Anthropic states that its bots, including Claude-User, honor robots.txt.
Action: decide on training (ClaudeBot) and on search visibility (Claude-SearchBot and Claude-User) as two separate questions.
How to Block or Allow ClaudeBot in robots.txt
To keep your site out of Claude's training data while staying available to Claude search and user requests:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
To slow ClaudeBot down instead of blocking it, Anthropic supports Crawl-delay; its own example is:
User-agent: ClaudeBot
Crawl-delay: 1
Two details from Anthropic's page are easy to miss. Rules apply per host, so you need them "for every subdomain that you wish to opt out from". And blocking Anthropic's IP addresses instead of using robots.txt "may not work correctly or persistently guarantee an opt-out". Anthropic also says it will not try to bypass CAPTCHAs.
A User-agent: * group with Disallow: / blocks all three bots unless they have groups of their own - including Claude-SearchBot, which removes you from Claude's search results. Our AI crawlers list shows the same pattern across other AI companies.
Action: check every subdomain that serves content, not only the main host, and make sure a wildcard group is not deciding Claude-SearchBot's access for you.
How to Verify That a Request Really Comes From Anthropic
Anthropic publishes the source addresses of its crawlers at claude.com/crawling/bots.json; on October 11, 2026 the list held 38 IPv4 prefixes, not labelled per bot. Anthropic's page says that if a crawler's source IP is on this list, "it indicates that the crawler is coming from Anthropic." Reverse DNS is not a usable check: the sample addresses we looked up belong to Amazon and Google Cloud and have no Anthropic hostname.
This list is new. In January 2025 Anthropic's page said it did not publish IP ranges, and in August 2025 Cloudflare wrote that Anthropic still lagged in verification, which made its crawler easier to spoof. To count Anthropic's bots in an access log:
grep -oEi 'ClaudeBot|Claude-SearchBot|Claude-User' access.log | sort | uniq -c
If you need to reach Anthropic about its crawler, the page asks you to write from an email address on the domain you are contacting them about.
Action: check suspicious "ClaudeBot" traffic against bots.json before blocking it; a spoofed crawler will not be in the list.
How Much Does ClaudeBot Crawl?
- ClaudeBot's share of AI and search crawling rose from 6% to nearly 10% between July 2024 and July 2025, and its share of AI-only crawling from 15% to 23.3% (Cloudflare, August 2025).
- Anthropic's crawl-to-refer ratio fell from about 286,900 to 1 in January 2025 to about 38,000 to 1 in July. Across 2025 it peaked at around 500,000 to 1 and later settled between roughly 25,000 and 100,000 to 1 (Cloudflare 2025 Year in Review). Cloudflare notes that Claude's native apps send no referrer, so these ratios likely overstate the gap.
- ClaudeBot was one of the three user agents most often fully disallowed in robots.txt in 2025, with GPTBot and CCBot.
- On Vercel's network, Claude's crawler made 370 million requests in one month of late 2024, and 34.16% of its fetches hit 404 pages (Vercel and MERJ, December 2024).
Action: if ClaudeBot load is the concern, try Crawl-delay or a path-level Disallow before blocking the whole site.
Does ClaudeBot Render JavaScript?
In Vercel and MERJ's December 2024 study, Claude's crawler fetched JavaScript files - 23.84% of its requests - but did not execute them. Content that only appears after client-side rendering is invisible to it. The study is from 2024 and predates Claude-SearchBot; our post on AI crawlers and JavaScript has the full matrix, and the JavaScript content gap check flags pages at risk.
Action: load your key pages with JavaScript disabled; that is roughly what ClaudeBot sees.
Should You Block ClaudeBot?
Blocking ClaudeBot is a training decision: it keeps your future content out of Anthropic's training data and does not change whether Claude can find your pages through search. If you want to appear in Claude's answers, keep Claude-SearchBot and Claude-User allowed. In our AI bot access check, a blocked ClaudeBot passes as an expected policy choice, while a blocked Claude-SearchBot is flagged because it removes your content from Claude search. For OpenAI's equivalent, see our guide to GPTBot.
Action: write your training and search decisions for Anthropic as separate groups, then run a free audit to confirm robots.txt says what you meant.
Looking for something else named ClaudeBot? Several unrelated open-source projects use the name, for example an IRC bot built on GitHub's Hubot. This article is about Anthropic's web crawler.
Sources
- Does Anthropic crawl data from the web, and how can site owners block the crawler? - Anthropic
- Anthropic crawler IP list (bots.json)
- AI Bots and Robots.txt - Paul Calvano, August 2025
- The crawl-to-click gap - Cloudflare, August 2025
- Cloudflare Radar 2025 Year in Review
- The rise of the AI crawler - Vercel and MERJ, December 2024
