llms.txt One Year Later: Who's Actually Reading It in 2026

Twenty months after Jeremy Howard proposed llms.txt on September 3, 2024, the file has the rare status of being widely advised by SEO consultants and almost entirely ignored by the systems it was supposed to serve. We downgraded llms.txt in our own audit back in March 2026 — moved it from a warning that cost the page score points to a quiet note that surfaces what we found and stops there. The downgrade was based on log-file analyses available at the time. The Q2 2026 evidence has now made the case stronger, not weaker.

This post walks through what changed, what the data actually says, and why we're not reversing the call.

Key Takeaways

• A 90-day server-log study by OtterlyAI (Feb 5, 2026) recorded 84 hits on /llms.txt out of 62,100+ AI bot requests — 0.1% of AI crawler traffic. The named major-citation crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended) did not show up in the file's access logs in meaningful numbers.
• SE Ranking's November 2025 analysis of ~300,000 domains found 10.13% adoption and no statistically significant correlation between having llms.txt and AI citation frequency. Removing the feature from their predictive model improved model accuracy.
• Google has said no on the record: Gary Illyes at Search Central Live and John Mueller in a separate Q&A both confirmed Google does not support llms.txt and has no plans to. Mueller compared it to the deprecated keywords meta tag.
• OpenAI's documented recommendation is to use robots.txt for crawler control, not llms.txt. Anthropic has not publicly committed ClaudeBot to consuming the file as a retrieval signal, even though Anthropic itself publishes llms.txt and llms-full.txt at docs.claude.com (which is a developer-tooling use of the file, not a search-citation one).
• Our audit no longer scores llms.txt presence as a ranking signal and no longer warns when it's missing. The file is still detected and surfaced as a neutral observation, and a quick format hint is offered if you do publish one — same pattern we apply to other "deprecated but observed" markup.

The Spec Has Done Its Job — Just Not the Job SEO Thinks It Did

Jeremy Howard's original proposal was specific: LLM context windows are too small for full sites, HTML-to-text conversion at retrieval time is error-prone, and developer documentation in particular suffers from being wrapped in navigation, ads, and JavaScript. A small markdown file at the site root, pointing at clean versions of the docs that matter, lets coding agents fetch the right context without parsing rendered HTML.

That use case works. Anthropic publishes llms.txt at docs.anthropic.com/llms.txt and a 481,349-token llms-full.txt at docs.claude.com/llms-full.txt. Mintlify bakes it into their docs platform. Stripe and Cloudflare publish their own. When Cursor, Windsurf, or Claude Code need to ground a coding answer in a vendor's current API, those clients can — and do — fetch the file directly because the user told them to.

The drift happened when the SEO industry took a file designed for developer-tooling agents and reinterpreted it as a search-discoverability signal. The renaming alone is telling: it is now routinely called "the robots.txt for AI" in client-facing collateral, even though the llmstxt.org spec makes no such claim and is explicit that the file is meant to help LLMs use a site once retrieval is already underway, not to influence whether the site gets cited in the first place.

Two different files, two different audiences, one shared filename. The developer-tooling job works. The search-citation job is the one we keep finding no evidence for.

What the 2026 Log Data Actually Shows

Three independent data points from Q1-Q2 2026 converge on the same finding.

90-day server-log study — OtterlyAI

The OtterlyAI experiment (published February 5, 2026) instrumented their own site with llms.txt and tracked AI-bot traffic for 90 days. Out of 62,100+ AI bot requests, just 84 touched /llms.txt — 0.1%. The only bot consistently accessing the file was BuiltWith, which catalogues web technology rather than answering user queries.

The named AI-search bots — GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended — were either absent from the /llms.txt access log or present at rates statistically indistinguishable from zero. The file received fewer AI visits than the average content page on the same site.

300,000-domain citation correlation — SE Ranking

SE Ranking's November 2025 study is the largest published analysis on the question. Sample size: roughly 300,000 domains. Method: correlate llms.txt presence with AI citation frequency across major LLM answers, then test whether including the feature improves a predictive model.

Headline numbers:

• 10.13% adoption across the sampled domains.
• Adoption distribution was nearly flat: 9.88% on low-traffic sites (0-100 visits), 10.54% on mid-traffic (1,001-5,000), 8.27% on the largest (100,001+). The biggest sites are slightly less likely to have the file than mid-tier ones — the opposite of what you'd expect if it were correlated with sophisticated SEO.
• No statistically significant correlation between llms.txt presence and citation frequency in any major AI answer surface.
• Removing llms.txt from the predictive feature set improved model accuracy. That's the strongest possible negative result short of a controlled before/after — the feature was actively dragging the model down.

The SE Ranking team's own framing: llms.txt "doesn't seem to directly impact AI citation frequency. At least not yet."

Vendor positions on the record

• Google — Gary Illyes at Search Central Live: Google does not support llms.txt and has no plans to. John Mueller separately, in a Q&A: "FWIW no AI system currently uses llms.txt" — comparing it to the deprecated keywords meta tag. Both quotes circulated through Search Engine Roundtable and Search Engine Land in late 2025 and have not been walked back.
• OpenAI — No public commitment to read llms.txt as a retrieval signal. OpenAI's documented crawler-control mechanism is robots.txt. GPTBot's access patterns on /llms.txt are at noise floor in every log analysis we have seen.
• Anthropic — Publishes llms.txt and llms-full.txt for its own documentation, but has not publicly confirmed that ClaudeBot consumes the file as part of inference or that Claude weights it as a retrieval signal. Claude Desktop and Claude.ai will fetch a user-supplied URL — including an /llms.txt URL — when a user pastes one. That is a user-initiated fetch, not a citation-shaping signal.
• Perplexity — Some retrieval activity reported in third-party guides, but no public official statement at the level Anthropic and Google have given. PerplexityBot's /llms.txt access in the OtterlyAI logs is in single digits.

The pattern across all four major providers is consistent: either explicit "no", silence, or developer-tooling use that doesn't generalize to search-citation lift. (Our companion post on how AI crawlers actually behave walks through the live user-agent landscape in more detail.)

Why the Original Downgrade Stands

We downgraded llms.txt from a warning to a neutral observation in March 2026, based on the log analyses then available. The post-March evidence — OtterlyAI's 90-day study, SE Ranking's 300,000-domain analysis, Mueller's "no AI system currently uses llms.txt" quote — has all pointed the same direction. None of the four scenarios that would have justified reversing the call have materialized:

1. A major crawler publicly committing to read the file. Didn't happen. Google said the opposite on the record. OpenAI and Anthropic have not made the commitment despite repeated press cycles.
2. A citation lift visible at population scale. Didn't happen. The 300,000-domain study found the opposite — including llms.txt in the feature set degraded predictive accuracy.
3. A meaningful fetch rate from named AI-search bots. Didn't happen. 0.1% in the only published 90-day log study; statistical noise for the named crawlers.
4. A spec evolution that gives the file a job AI engines couldn't get from HTML. Didn't happen. The spec is stable; AI engines extract from rendered HTML during retrieval, which both searchVIU's experiments and the broader log evidence confirm.

So our policy stays where it has been since March: the audit detects llms.txt when it's present, lists the format issues that would matter if you publish one (no # H1 title, no ## sections, no descriptive paragraph, no markdown links), and stops there. No score impact either way. The check exists because publishing a usable llms.txt is cheap if you choose to do it, and because some teams do use it productively for the developer-tooling case Howard originally described — we want them to be able to verify their file is well-formed without us pretending the file moves their search rank.

What We're Not Saying

A few precise distinctions, because this is the kind of post that attracts straw-man counterarguments.

We are not saying llms.txt is useless. For developer documentation consumed by coding agents (Claude Code, Cursor, Windsurf, Continue), llms.txt is genuinely useful as a curated index the agent can fetch on demand. If your audience is developers building against your API and you maintain llms.txt and llms-full.txt for that purpose, keep doing it.

We are not saying you should remove existing llms.txt files. They are cheap to keep. The cost is in prioritizing the work — putting llms.txt on a content roadmap, paying a consultant to "optimize" it, treating it as a remediation item on an audit. The maintenance hours are real and the SEO outcome is not.

We are not saying no AI system has ever fetched /llms.txt. Some do, occasionally. Anthropic's own documentation discovery probably uses it. Specialized agents pointed at the file by a user will fetch it. The claim we're evaluating is narrower and concrete: does presence of llms.txt change the rate at which your pages get cited in mainstream AI answers (ChatGPT, Claude, Perplexity, Google AI Overviews)? The 2026 evidence says no.

We are not making a claim about 2027. If OpenAI or Anthropic ship a retrieval pipeline that reads llms.txt and the citation evidence flips, we will re-score. The check exists, the file is detected, the data hooks are in place. Re-weighting would be a single config change.

A Practical 5-Minute Decision Tree

If you're trying to decide what to do about llms.txt for your own site, the answer is short:

1. Are you publishing developer-facing technical documentation that coding agents will consume? If yes — publish llms.txt and llms-full.txt. Keep them current. The use case works.
2. Do you currently have llms.txt deployed and someone is paying for ongoing optimization of it? Stop the optimization spend. The file is fine to keep; the ongoing tuning is not producing measurable outcomes.
3. Are you about to add llms.txt as part of an SEO remediation plan? Defer. The Q2 2026 evidence says it won't move citation rates. Spend the hours on the signals that do — original research, firsthand experience patterns, comparison tables, properly structured H2/H3 hierarchies. Our previous posts on original research as an AI citation signal and what AI search engines actually check walk through the alternatives.
4. Is an audit tool warning you that missing llms.txt is costing your score? Get a different audit tool. Or wait — most of them will catch up to the 2026 evidence in the next quarter.

The honest answer to "should I add llms.txt for AI search visibility in mid-2026?" is no, not for that reason. The honest answer to "is llms.txt a meaningful AI ranking signal?" is not in the data we have, and the data is now substantial enough to act on.

What This Means for the Broader "AI SEO" Stack

llms.txt is the second piece of "AI-first" markup to fail the Q2 2026 evidence test in a row. We covered the FAQ rich result deprecation two weeks ago: FAQ schema lost its Google rich result on May 7 and Ahrefs' 1,885-page controlled study found no AI citation lift. llms.txt is now in the same category — a piece of optional markup that the industry sold as a ranking lever and that the data shows isn't one.

The pattern matters more than either individual file. Both were proposed as cheap technical optimizations that would produce a visibility outcome. Both got enthusiastic adoption from teams looking for low-cost wins. Both have now run for long enough to be measured at population scale. Neither produces the outcome it was sold as producing.

The signals that do show up in the post-April-Core-Update analyses — original research, firsthand experience, demonstrably author-produced photos and measurements, properly structured content — all share one property: they cost real human effort to produce. The trade-off is no longer "do the cheap markup work and the algorithm will reward you." It's "do the work the algorithm can detect because the work happened, and the markup is decoration."

Our audit's job is to score against the signals that the evidence supports. When the evidence moves, the weights move. When we find a check that the data no longer justifies, we say so out loud — that's why this post exists, and why the March policy change on FAQ schema looked the same in shape: published evidence, published weight change.

Want to see how your pages score against the May 2026 weights — including the deprecated llms.txt and FAQ-schema signals, and the upgraded Experience and original-research ones? Run a free audit at hybridranking.com.

Sources

1. The /llms.txt proposal — Jeremy Howard, Answer.AI (Sept 3, 2024)
2. The /llms.txt spec — llmstxt.org
3. The llms.txt Experiment, 90-day log study — OtterlyAI (Feb 5, 2026)
4. LLMs.txt Shows No Clear Effect On AI Citations (300k domains) — Search Engine Journal / SE Ranking (Nov 20, 2025)
5. Google says normal SEO works for ranking in AI Overviews and llms.txt won't be used — Search Engine Land
6. Simplifying docs for AI with /llms.txt — Mintlify
7. Anthropic's published llms.txt — docs.anthropic.com
8. Schema markup AI-citation study, 1,885 pages — Ahrefs (May 2026)