What Is GPTBot? OpenAI's Crawler, What It Does and How to Control It
GPTBot is the web crawler OpenAI uses to collect content that may be used to train its AI models. It is also the AI crawler site owners block most often, according to Cloudflare's 2025 review of robots.txt files: the 2025 Web Almanac found GPTBot named in 4.5% of robots.txt files, up from 2.9% a year earlier, and an August 2025 analysis by Paul Calvano found GPTBot rules on almost 21% of the top 1,000 websites.
Many of those rules rest on a misunderstanding. Blocking GPTBot does not remove a site from ChatGPT's answers. OpenAI runs four separate agents, and the one that decides whether you can appear in ChatGPT search is OAI-SearchBot, not GPTBot. This guide explains what GPTBot does, how it differs from OpenAI's other bots, and how to control each one deliberately.
Key Takeaways
- GPTBot is a training crawler. OpenAI says it crawls content "that may be used in training our generative AI foundation models".
- It is one of four OpenAI agents. OAI-SearchBot powers ChatGPT search, ChatGPT-User fetches pages users ask about, and OAI-AdsBot checks ad landing pages.
- The robots.txt settings are independent. OpenAI's own example: allow OAI-SearchBot to appear in search results while disallowing GPTBot.
- Its crawling grew fast. Cloudflare measured GPTBot's share of AI and search crawling rising from 4.7% in July 2024 to 11.7% in July 2025.
- It reads raw HTML only. In Vercel's December 2024 measurements GPTBot did not execute JavaScript, so content that only appears after client-side rendering is invisible to it.
What Is GPTBot?
GPTBot is OpenAI's crawler for model training. According to OpenAI's crawler documentation, "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." Its user agent currently looks like this:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
OpenAI notes that the version number may change, so match on the GPTBot token rather than the full string. When GPTBot fetches robots.txt itself, OpenAI may add a robots.txt marker to the user agent, which helps separate those requests from page fetches in your logs.
Action: write any rule as User-agent: GPTBot, never against a full user-agent string.
GPTBot vs OAI-SearchBot vs ChatGPT-User vs OAI-AdsBot
OpenAI documents four agents. Each is controlled separately:
| Agent | What it does | robots.txt | IP list |
|---|---|---|---|
GPTBot |
Collects content that may be used to train OpenAI's models | Honored | gptbot.json |
OAI-SearchBot |
Builds the index for ChatGPT search; opted-out sites "will not be shown in ChatGPT search answers" | Honored | searchbot.json |
ChatGPT-User |
Visits a page when a ChatGPT user asks about it; "not used for crawling the web in an automatic fashion" | "may not apply" | chatgpt-user.json |
OAI-AdsBot |
Checks pages submitted as ChatGPT ads; data "is not used to train" models | Not stated | adsbot.json |
Two details from the same documentation matter in practice. First, "Each setting is independent of the others." Second, if you allow both GPTBot and OAI-SearchBot, OpenAI "may use the results from just one crawl for both use cases", so you will not necessarily see two sets of requests. A robots.txt change takes about 24 hours to reach ChatGPT search.
Action: list all four agents in your robots.txt policy notes, even if you only write rules for two of them.
Does Blocking GPTBot Remove You From ChatGPT?
No. Blocking GPTBot tells OpenAI not to use your content for training. Whether your pages can be shown and cited in ChatGPT search answers is controlled by OAI-SearchBot. OpenAI gives exactly this combination as an example: allow OAI-SearchBot in order to appear in search results while disallowing GPTBot.
The mistake that does remove sites from ChatGPT is broader: a User-agent: * group with Disallow: /, written to stop "AI training", also blocks OAI-SearchBot when OAI-SearchBot has no group of its own. Our ChatGPT SEO guide covers the search side in detail.
Action: if you block GPTBot, add an explicit group for OAI-SearchBot so the search decision is visible, not inherited.
How to Block or Allow GPTBot in robots.txt
To keep your whole site out of training while staying eligible for ChatGPT search:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
To keep only part of the site out of training, for example a members area:
User-agent: GPTBot
Disallow: /members/
To block several training crawlers with one rule, list them together; consecutive User-agent lines share the rules below them:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
robots.txt is a request, not an access control. For the user-triggered ChatGPT-User, OpenAI says robots.txt rules "may not apply", because a person started the request. If you must keep those requests out, use the published IP list at your firewall. Our AI crawlers list shows how the other AI companies handle the same distinction, and our ClaudeBot guide covers Anthropic's three bots.
Action: after editing robots.txt, check for a leftover User-agent: * / Disallow: / that overrides what you meant for OAI-SearchBot.
How to Verify That a Request Really Comes From GPTBot
Anyone can send a request with "GPTBot" in the user agent. OpenAI publishes the IP ranges its crawlers use: gptbot.json held 18 IPv4 prefixes on October 11, 2026. OpenAI does not document reverse-DNS verification, so the check is the user-agent token plus a match against that list. OpenAI also publishes a key directory for signed requests at chatgpt.com/.well-known/http-message-signatures-directory, which services such as Cloudflare use to verify OpenAI agents.
To count OpenAI's agents in a standard access log:
grep -oEi 'GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot' access.log | sort | uniq -c
Action: before blocking "GPTBot" traffic that looks abusive, check the source IPs against gptbot.json; a spoofed crawler will not be in it.
How Much Does GPTBot Crawl?
Cloudflare's network data gives the clearest picture:
- GPTBot's share of AI and search crawling rose from 4.7% in July 2024 to 11.7% in July 2025, and its share of AI-only crawling from 11.9% to 28.1% (Cloudflare, August 2025).
- In July 2025 OpenAI's crawlers fetched about 1,091 pages for every visitor ChatGPT referred, down from 1,217 in January. The ratio peaked at around 3,700 to 1 in March 2025 (Cloudflare 2025 Year in Review).
- GPTBot generated about 7.5% of verified bot traffic in 2025, and was one of the three user agents most often fully disallowed in robots.txt files, with ClaudeBot and CCBot.
- On Vercel's network, GPTBot made 569 million requests in one month of late 2024, about an eighth of Googlebot's 4.5 billion (Vercel and MERJ, December 2024).
Action: if GPTBot load is a problem, check your logs for which paths it fetches before blocking the whole site; a path-level Disallow may be enough.
Does GPTBot Render JavaScript?
In Vercel and MERJ's December 2024 study, none of OpenAI's crawlers executed JavaScript. They downloaded JavaScript files - 11.50% of ChatGPT's requests - but did not run them. Content that only appears after client-side rendering is therefore invisible to GPTBot and to OAI-SearchBot. That study is from 2024; we have not seen newer measurements that contradict it. Our post on AI crawlers and JavaScript covers the wider crawler set, and our JavaScript content gap check flags pages at risk.
Action: view your page with JavaScript disabled; whatever is missing there is missing for GPTBot.
Should You Block GPTBot?
It is a content-policy decision. Blocking GPTBot keeps your future content out of OpenAI's training data; allowing it lets OpenAI use that content to train models. Neither choice changes whether ChatGPT search can show your pages - that depends on OAI-SearchBot. A common 2026 setup blocks the training crawlers and allows the search crawlers, so a site stays citable in AI answers without contributing to training.
Our audit treats it the same way. In the AI bot access check, a blocked GPTBot passes as an expected policy choice, while a blocked OAI-SearchBot is flagged because it removes the page from ChatGPT search.
Action: decide training and search separately, write both decisions into robots.txt, and run a free audit to confirm the file does what you intended.
