What Is PerplexityBot? Perplexity's Crawler, robots.txt and How to Verify It
PerplexityBot is the crawler Perplexity uses to find pages and link them in its search answers. Unlike GPTBot or ClaudeBot, it is not a training crawler: Perplexity's crawler documentation says it "is not used to crawl content for AI foundation models." Blocking it is therefore not a decision about training data. It is a decision about whether Perplexity can find your pages at all.
Perplexity is also the AI company whose crawling has been disputed most publicly. In August 2025 Cloudflare reported that sites which had blocked both of Perplexity's declared agents were still being accessed by an undeclared browser, and removed Perplexity from its list of verified bots; Perplexity disputed the attribution. This guide covers what each Perplexity agent does, how to control it, and how to tell a real PerplexityBot request from an impersonation.
Key Takeaways
- PerplexityBot is a search crawler, not a training crawler. Perplexity says it is "designed to surface and link websites in search results on Perplexity."
- Perplexity-User is separate and generally ignores robots.txt. It fetches pages when a user asks a question; Perplexity says it "generally ignores robots.txt rules."
- Perplexity recommends allowing PerplexityBot in robots.txt and permitting its published IP ranges if you want to appear in its search results.
- Verification is user agent plus IP. Perplexity publishes IP lists in JSON and does not document reverse-DNS verification.
- It reads raw HTML. In Vercel and MERJ's December 2024 study, PerplexityBot did not execute JavaScript.
What Is PerplexityBot?
PerplexityBot crawls the web to build the index that Perplexity searches when it answers a question. Its user agent, as documented by Perplexity:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Perplexity's advice for site owners is direct: "To ensure your site appears in search results, we recommend allowing PerplexityBot in your site's robots.txt file and permitting requests from our published IP ranges." A robots.txt change "may take up to 24 hours" to reach Perplexity's systems.
Action: write rules against the PerplexityBot token, not the full user-agent string, so a version change does not break them.
PerplexityBot vs Perplexity-User
Perplexity documents two agents, and they follow different rules:
| Agent | What it does | robots.txt | IP list |
|---|---|---|---|
PerplexityBot |
Crawls pages to "surface and link websites in search results on Perplexity"; not used for model training | The control Perplexity documents | perplexitybot.json |
Perplexity-User |
Visits a page when a user's question needs it, and may link it in the answer; "not used for web crawling" or training | "generally ignores robots.txt rules" | perplexity-user.json |
Perplexity's reasoning for the second row is that a person started the request: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." OpenAI says something similar about ChatGPT-User, as covered in our GPTBot guide. In practice this means Disallow stops PerplexityBot from indexing a page, but does not stop Perplexity from fetching it when a user asks about it.
Action: treat robots.txt as the control for PerplexityBot only; if you need to keep Perplexity-User out, that takes a firewall rule.
How to Allow or Block PerplexityBot in robots.txt
To allow Perplexity's search crawler explicitly:
User-agent: PerplexityBot
Allow: /
To keep it out of one section, use a path rule instead of blocking the whole site:
User-agent: PerplexityBot
Disallow: /internal/
The common accident is a site-wide User-agent: * / Disallow: / group written to stop AI crawlers. A crawler with no group of its own follows the * group, so PerplexityBot is blocked along with everything else. Our AI crawlers list covers the other search crawlers caught the same way, and the conflicting rules check flags robots.txt groups that contradict each other.
robots.txt is not the only gate. Perplexity notes that "If you're using a Web Application Firewall (WAF) to protect your site, you may need to explicitly whitelist Perplexity's bots," and gives steps for Cloudflare and AWS WAF that combine a user-agent match with its IP ranges. For AWS WAF it adds: "ensure they have higher priority than blocking rules."
Action: if you want to appear in Perplexity, give PerplexityBot its own Allow group and check that no bot-protection rule answers it with a 403.
Does Perplexity Respect robots.txt?
Perplexity's documentation presents robots.txt as the control: "Webmasters can use the following robots.txt tags to manage how their sites and content interact with Perplexity." The dispute is about what happens when a site uses them.
Cloudflare's August 4, 2025 report described a test on newly purchased domains that "had not yet been indexed by any search engine." The sites blocked "both of Perplexity's declared crawlers: PerplexityBot and Perplexity-User", yet Perplexity could still answer questions about their content. Cloudflare attributed this to "a generic browser intended to impersonate Google Chrome on macOS", sending 3 to 6 million requests a day from IP addresses outside Perplexity's published ranges and from rotating networks. For comparison, the declared Perplexity-User agent made 20 to 25 million requests a day. Cloudflare responded by de-listing Perplexity as a verified bot and adding the undeclared crawler's signatures to its AI-crawler managed rule, which it made available to free customers too.
Perplexity's response disputed the attribution: it said Cloudflare "fundamentally misattributed 3-6M daily requests" from BrowserBase, a third-party cloud browser service, and that its own use of that service was "less than 45,000 daily requests." It also defended user-triggered fetching in general, comparing it to Google's user-triggered fetchers.
We cannot settle the attribution from the outside. What both sides agree on is that robots.txt controls PerplexityBot, while user-triggered fetches follow different rules.
Action: if you need Perplexity kept out entirely, rely on a firewall or CDN rule, not on robots.txt alone.
How to Verify That a Request Really Comes From PerplexityBot
Anyone can put "PerplexityBot" in a user agent. Perplexity's guidance is to "combine both User-Agent string matching and IP address verification", using "the most current IP ranges from the official JSON endpoints." On October 12, 2026, perplexitybot.json listed 8 IPv4 prefixes and perplexity-user.json listed 4. Perplexity does not document reverse-DNS verification, so the IP lists are the check.
To count Perplexity's agents in a standard access log:
grep -oEi 'PerplexityBot|Perplexity-User' access.log | sort | uniq -c
Action: refresh the IP lists automatically, as Perplexity recommends, rather than copying the prefixes into a config file once.
How Much Does PerplexityBot Crawl?
Compared with the large AI crawlers, not much, and it sends more visitors per page crawled:
- PerplexityBot was 0.4% of AI-only crawling in July 2025, up from near zero a year earlier (Cloudflare, August 2025). GPTBot, for comparison, was 28.1%.
- In the same data, Perplexity crawled 194.8 pages for every visitor it referred in July 2025, up from 54.6 in January. OpenAI's ratio was 1,091 to 1.
- Over all of 2025, Perplexity "had the lowest crawl-to-refer ratios of the major AI platforms", though the ratio spiked above 700 to 1 in late March before falling below 200 to 1 from September (Cloudflare 2025 Year in Review).
- On Vercel's network, PerplexityBot made 24.4 million fetches in one month of late 2024 (Vercel and MERJ, December 2024).
Action: if Perplexity traffic shows up in your logs, compare it with referrals from perplexity.ai in your analytics before deciding it costs more than it returns.
Does PerplexityBot Render JavaScript?
No, as of the last published measurement. Vercel and MERJ found that "none of the major AI crawlers currently render JavaScript", and listed PerplexityBot among them. That study is from December 2024; we have not found newer measurements. Content that only appears after client-side rendering is invisible to PerplexityBot, which our post on AI crawlers and JavaScript covers in detail. The JavaScript content gap check flags pages at risk.
Action: load your key pages with JavaScript disabled; what is missing there is missing for PerplexityBot.
Should You Block PerplexityBot?
Only if you do not want to appear in Perplexity. Because PerplexityBot is not a training crawler, the common "block training, allow search" setup leaves it allowed. Perplexity does not state in its documentation that a blocked site disappears from its answers, but it recommends allowing the crawler "to ensure your site appears in search results", and in Cloudflare's test, answers about blocked sites "were less specific and lacked details from the original content" once the block worked.
Our audit follows the same split. In the AI bot access check, a blocked PerplexityBot is flagged because it also blocks AI search visibility, while a blocked Perplexity-User is flagged with a note that Perplexity says robots.txt may not apply to it. For the equivalent decisions at OpenAI and Anthropic, see our guides to GPTBot and ClaudeBot.
Action: allow PerplexityBot unless you have a reason to stay out of Perplexity, decide separately about Perplexity-User at the firewall, and run a free audit to see how your robots.txt treats each agent.
Sources
- Perplexity Crawlers - Perplexity documentation
- Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives - Cloudflare, August 2025
- Agents or Bots? Making Sense of AI on the Open Web - Perplexity
- The crawl-to-click gap - Cloudflare, August 2025
- Cloudflare Radar 2025 Year in Review - Cloudflare, December 2025
- The rise of the AI crawler - Vercel and MERJ, December 2024
