What Is Bytespider? ByteDance's Crawler, robots.txt and How to Verify It
Bytespider is the web crawler run by ByteDance, the company that owns TikTok. In mid-2024 it was the busiest AI crawler on Cloudflare's network: Cloudflare named it first among "the top four AI crawlers" and found it on 40.40% of the websites it measured. A year later its share had collapsed, and a 2026 measurement found it ignoring robots.txt more often than any other bot it measured except ChatGPT-User.
It is also poorly documented in English. Some AI crawler lists, including ours until this week, said ByteDance publishes nothing about it. That is not quite right: ByteDance's Toutiao Search webmaster platform has a Chinese-language page about Bytespider with its user agents, IP ranges and a verification method. This guide covers what that page says, what it leaves out, and how to control the crawler.
Key Takeaways
- ByteDance documents Bytespider as a search crawler. Its Toutiao Search webmaster page names Bytespider as Toutiao Search's crawler and does not mention AI training.
- Training use is reported, not documented. Cloudflare says Bytespider "is reportedly used to gather training data" for ByteDance's large language models.
- Its robots.txt compliance is in doubt. In TollBit's analysis of European sites in the first half of 2026, Bytespider ignored explicit robots.txt instructions on 48% of its scrapes.
- It can be verified by reverse DNS. ByteDance says a genuine Bytespider hostname ends in bytedance.com and anything else is an impersonator.
- Its crawling fell sharply. Its share of AI and search crawler traffic dropped from 14.1% in July 2024 to 2.4% in July 2025, according to Cloudflare.
What Is Bytespider?
ByteDance's page states that the crawler user agent of Toutiao Search, its search engine, is "Bytespider", with a capital B. The page describes a search engine's steps - crawling, indexing and serving results - and lists three user agents, for desktop, Android and iOS. The desktop one:
Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36
The Android and iOS strings carry the same (compatible; Bytespider; https://zhanzhang.toutiao.com/) marker. The page is in Chinese and undated, which helps explain why English-language crawler lists miss it.
Action: write rules and log filters against the Bytespider token, which all three user agents share.
Is Bytespider an AI Crawler?
ByteDance does not say so. Its page describes Bytespider only as a search crawler. The AI label comes from third parties. Cloudflare's July 2024 analysis says that "Bytespider is reportedly used to gather training data" for ByteDance's large language models, including those behind its chatbot Doubao, and Cloudflare's 2024 Year in Review calls it "ByteDance's AI crawler", "reportedly used to download training data for ByteDance's Large Language Models (LLMs)". We have not found a ByteDance statement that confirms or denies this.
So the honest answer is: officially a search crawler, reported to collect training data too. Our own AI crawlers list groups it with the training crawlers for that reason.
Action: treat a decision about Bytespider as two decisions at once: whether you want to appear in Toutiao Search, and whether you accept the reported training use.
How to Block or Allow Bytespider in robots.txt
To block it across the whole site:
User-agent: Bytespider
Disallow: /
The Bytespider page itself says nothing about robots.txt. A separate page on the same platform explains how Toutiao Search matches robots.txt paths and points to Google's rules for the details. Neither page states that Bytespider obeys a Disallow.
If your problem is load rather than access, ByteDance offers another route: a crawl-frequency setting in its webmaster platform, which it says takes effect within one day.
A site-wide User-agent: * / Disallow: / group also covers Bytespider when it has no group of its own. Our conflicting rules check flags robots.txt groups that contradict each other.
Action: if you block Bytespider, give it a named group so the decision is explicit, and confirm in your logs that the requests actually stop.
Does Bytespider Respect robots.txt?
The measurements say often not. TollBit's State of the Bots report for the first half of 2026 in its analysis of European sites, defines the bypass rate as "the percentage of all scrapes that ignored explicit robots.txt guidelines and successfully scraped content anyway". ChatGPT-User had the highest, at 54%, "followed by Bytespider with a bypass rate of 48%". The same report says that "ByteDance has no published robots.txt policy."
Cloudflare's 2024 data shows the other side of the problem: its customers "do not specifically disallow popular AI crawlers like Bytespider and ClaudeBot", even though Bytespider led in requests at the time.
Action: if you need Bytespider kept out, enforce it at the firewall or CDN by user agent or IP range; robots.txt alone states your preference but did not stop about half of its scrapes in TollBit's data.
How to Verify That a Request Really Comes From Bytespider
Anyone can send "Bytespider" in a user agent. ByteDance's page gives two checks. First, reverse DNS: a genuine Bytespider hostname ends in .bytedance.com, and the page says any other hostname is an impersonator. Second, an IP list of 10 ranges:
110.249.201.0/24 110.249.202.0/24
111.225.148.0/24 111.225.149.0/24
220.243.135.0/24 220.243.136.0/24
220.243.188.0/24 220.243.189.0/24
60.8.123.0/24 60.8.151.0/24
We tested the reverse lookup on October 12, 2026. For 111.225.149.10 it returned bytespider-111-225-149-10.crawl.bytedance.com, as described. The usual second step, resolving that hostname forward to confirm it points back to the same IP, failed: the hostname returned no address. Use reverse DNS together with the IP list instead.
dig +short -x 111.225.149.10
grep -c 'Bytespider' access.log
Action: check suspicious Bytespider requests against both the hostname suffix and the 10 ranges, and treat a request that fails either as an impersonator.
How Much Does Bytespider Crawl?
Much less than it used to:
- In July 2024 Cloudflare called Bytespider, Amazonbot, ClaudeBot and GPTBot "the top four AI crawlers", with Bytespider leading in requests and reaching 40.40% of the websites it measured.
- By the end of November 2024, its activity was "approximately 80-85% lower than that seen at the start of the year" (Cloudflare 2024 Year in Review).
- Its share of AI and search crawler traffic fell from 14.1% in July 2024 to 2.4% in July 2025; among AI-only crawlers it fell from 37.3% to 5.8% (Cloudflare, August 2025).
- Through 2025 its "activity dropped across the year, continuing the decline observed last year" (Cloudflare 2025 Year in Review).
- In the 2025 Web Almanac data, 1.3% of robots.txt files named Bytespider as a user agent, about 204,000 mobile sites.
Action: before writing rules, count Bytespider in your own logs; at today's volumes it may be a minor part of your bot traffic.
Does Bytespider Render JavaScript?
No, as of the last published measurement. Vercel and MERJ found that "none of the major AI crawlers currently render JavaScript" and named ByteDance's Bytespider among them (Vercel and MERJ, December 2024). Content that only appears after client-side rendering is invisible to it. Our post on AI crawlers and JavaScript has the full matrix, and the JavaScript content gap check flags pages at risk.
Action: if you allow Bytespider, make sure your important content is in the initial HTML.
Should You Block Bytespider?
Blocking it removes you from a crawler ByteDance documents for Toutiao Search, and, by the reports above, from a source of training data for its models. If you do not want either, block it, and back the robots.txt rule with a firewall rule, given the bypass data. If you want visibility in Toutiao Search, allow it and verify the traffic.
In our AI bot access check, Bytespider is scored with the training crawlers, so blocking it carries no score penalty. For the crawlers whose operators document their purposes in detail, see our guides to GPTBot, ClaudeBot and PerplexityBot.
Action: decide on Bytespider deliberately, enforce the decision where robots.txt is not enough, and run a free audit to see how your robots.txt treats it and the other AI crawlers.
Sources
- 关于Bytespider (About Bytespider) - Toutiao Search Webmaster Academy, ByteDance
- Robots匹配说明 (robots.txt matching) - Toutiao Search Webmaster Academy, ByteDance
- Declare your AIndependence: block AI bots, scrapers and crawlers with a single click - Cloudflare, July 2024
- Cloudflare Radar 2024 Year in Review - Cloudflare, December 2024
- The crawl-to-click gap - Cloudflare, August 2025
- Cloudflare Radar 2025 Year in Review - Cloudflare, December 2025
- State of the Bots, 2026 Q1 & Q2 - TollBit
- SEO chapter - 2025 Web Almanac, HTTP Archive
- The rise of the AI crawler - Vercel and MERJ, December 2024
