Sign in Sign up free

llms.txt One Year Later: Who's Actually Reading It in 2026

AI SEO Intelligence

May 25, 2026
Updated October 11, 2026
12 min read
llms.txt One Year Later: Who's Actually Reading It in 2026

Update, October 11, 2026: this post now covers spec v2 (August 2026), Google's July 2026 guidance, and two 2026 studies (Ahrefs' May 2026 logs and Common Crawl's July 2026 crawl). We also corrected several details: where and when Google's statements were made, which bots the OtterlyAI study names, and Anthropic's documentation URLs. The conclusion is unchanged.

Twenty months after Jeremy Howard proposed llms.txt in September 2024, the file has the rare status of being widely advised by SEO consultants and almost entirely ignored by the systems it was supposed to serve. We downgraded llms.txt in our own audit back in March 2026 - moved it from a warning that cost the page score points to a quiet note that surfaces what we found and stops there. The downgrade was based on log-file analyses available at the time. The Q2 2026 evidence has now made the case stronger, not weaker.

This post walks through what changed, what the data actually says, and why we're not reversing the call.

Key Takeaways

  • OtterlyAI's server-log experiment counted 84 AI bot visits to /llms.txt out of 62,100+ AI bot visits - about 0.1%. Ahrefs' June 2026 study of 137,210 domains found that 97% of llms.txt files got no traffic at all in May 2026.
  • SE Ranking's November 2025 analysis of ~300,000 domains found 10.13% adoption and no statistically significant correlation between having llms.txt and AI citation frequency. Removing the feature from their predictive model improved model accuracy.
  • Google says it in its own documentation: the AI optimization guide (updated July 10, 2026) tells site owners they can ignore "unnecessary AI text files (like llms.txt)" because Google Search does not use them. Gary Illyes said in July 2025 that Google does not support llms.txt and is not planning to.
  • OpenAI's crawler documentation covers robots.txt and does not mention llms.txt. Anthropic has not said its crawlers read the file, even though it publishes llms.txt and llms-full.txt for its own developer docs (a developer-tooling use, not a search-citation one).
  • Our audit no longer scores llms.txt presence as a ranking signal and no longer warns when it's missing. The file is still detected and surfaced as a neutral observation, and a quick format hint is offered if you do publish one - same pattern we apply to other "deprecated but observed" markup.

What Is llms.txt?

llms.txt is a plain markdown file at the root of a site (/llms.txt) that gives language models a short, curated map of the site: what it is, and which pages hold the information worth reading. Jeremy Howard proposed it on September 3, 2024; the spec lives at llmstxt.org and the reference repository is maintained by Answer.AI. Version 2 of the spec was published in August 2026. A file can also sit under a subpath such as /docs/llms.txt, in which case it covers the URLs below that path, and an agent should use the most specific file.

The spec describes the intended use plainly: the file "stays small enough to fit in context" and is read "on demand, when an agent needs information about a topic while assisting a user". It also notes that the format is "used most heavily for software documentation".

It is not a crawler-control file, and it is not a sitemap:

File What it does Who reads it
robots.txt Tells crawlers which URLs they may fetch Search engines and AI crawlers, by standard
sitemap.xml Lists every URL you want indexed, with dates Search engines
llms.txt A short, hand-picked reading list in markdown, with a one-line note per link Agents that choose to fetch it, mostly coding tools

llms.txt has no allow or disallow rules and no effect on crawling. It is a reading list for an agent that is already looking at your site, which is why the evidence below matters: the question is not whether the format works, but whether any AI search system reads it.

Adoption is real, even if the payoff is not. Common Crawl analysed 584,107 llms.txt files from its July 2026 crawl: only 49.90% had the complete shape the spec asks for, 22.56% contained no links at all, and 41.34% were generated by Wix. Ahrefs puts adoption at up to 28% of the domains in its sample, and calls that figure an upper bound.

The Spec Has Done Its Job - Just Not the Job SEO Thinks It Did

The original llmstxt.org proposal was specific: LLM context windows are too small for full sites, HTML-to-text conversion at retrieval time is error-prone, and developer documentation in particular suffers from being wrapped in navigation, ads, and JavaScript. A small markdown file at the site root, pointing at clean versions of the docs that matter, lets coding agents fetch the right context without parsing rendered HTML.

That use case works. Anthropic publishes llms.txt for its developer platform, plus an llms-full.txt of about 43 MB. OpenAI publishes one for its developer docs, and Perplexity does the same for its API docs. Mintlify generates the files for every docs site on its platform. When a coding agent needs to ground an answer in a vendor's current API, it can fetch the file directly because the user pointed it there.

The drift happened when the SEO industry took a file designed for developer-tooling agents and reinterpreted it as a search-discoverability signal. The renaming alone is telling: it is now routinely called "the robots.txt for AI" in client-facing collateral, even though the llmstxt.org spec makes no such claim. It describes a file read on demand by an agent that is already helping a user, not a signal that decides whether a site gets cited in the first place.

Two different files, two different audiences, one shared filename. The developer-tooling job works. The search-citation job is the one we keep finding no evidence for.

What the Log Data Actually Shows

Four independent data sources, from late 2025 to mid-2026, converge on the same finding.

Server-log experiment - OtterlyAI

The OtterlyAI experiment (page last updated February 5, 2026) published llms.txt on its own site and tracked AI-bot traffic. Out of 62,100+ AI bot visits, 84 went to /llms.txt - about 0.1%. The file received far fewer AI visits than the site's content pages.

May 2026 logs across 137,210 domains - Ahrefs

Ahrefs' study (June 15, 2026) is the largest log-based look so far. Of the domains that published the file, 97% saw zero requests to it in May 2026. AI retrieval bots made 1.1% of the requests that did happen. GPTBot was the most frequent requester, followed by Claude Code - a coding agent, which fits the developer-tooling use case. Ahrefs' 28% adoption figure for the sample is, in its own words, an upper bound.

300,000-domain citation correlation - SE Ranking

SE Ranking's November 2025 study is the largest published analysis on the question. Sample size: roughly 300,000 domains. Method: correlate llms.txt presence with AI citation frequency across major LLM answers, then test whether including the feature improves a predictive model.

Headline numbers:

  • 10.13% adoption across the sampled domains.
  • Adoption distribution was nearly flat: 9.88% on low-traffic sites (0-100 visits), 10.54% on mid-traffic (1,001-5,000), 8.27% on the largest (100,001+). The biggest sites are slightly less likely to have the file than mid-tier ones - the opposite of what you'd expect if it were correlated with sophisticated SEO.
  • No statistically significant correlation between llms.txt presence and citation frequency in any major AI answer surface.
  • Removing llms.txt from the predictive feature set improved model accuracy. That's the strongest possible negative result short of a controlled before/after - the feature was actively dragging the model down.

The SE Ranking team's own framing: llms.txt "doesn't seem to directly impact AI citation frequency. At least not yet."

Vendor positions on the record

  • Google - The clearest "no" comes from Google's own AI optimization guide (updated July 10, 2026): you can ignore "creating unnecessary AI text files (like llms.txt)", because Google Search itself does not use them, and doing so "will neither harm nor help". Before that, Gary Illyes said at a Search Central Deep Dive event in Asia-Pacific that Google does not support llms.txt and is not planning to (reported July 24, 2025). John Mueller compared the file to the keywords meta tag on Reddit (April 2025) and wrote on Bluesky in June 2025 that "no AI system currently uses llms.txt" (Search Engine Roundtable). Chrome's Lighthouse now has an llms.txt audit, but it only flags server errors; a missing file is marked not applicable.
  • OpenAI - No public statement about reading llms.txt. Its crawler documentation covers robots.txt only. In Ahrefs' May 2026 logs GPTBot was the most frequent requester of llms.txt files, yet 97% of files received no requests at all.
  • Anthropic - Publishes llms.txt and llms-full.txt for its own documentation, but its crawler page talks only about robots.txt and does not say ClaudeBot reads the file. Claude will fetch an /llms.txt URL when a user pastes one. That is a user-initiated fetch, not a citation-shaping signal.
  • Perplexity - No official statement. Its crawler documentation covers robots.txt only.

The pattern across all four major providers is consistent: either explicit "no", silence, or developer-tooling use that doesn't generalize to search-citation lift. (Our companion post on how AI crawlers actually behave walks through the live user-agent landscape in more detail.)

Why the Original Downgrade Stands

We downgraded llms.txt from a warning to a neutral observation in March 2026, based on the evidence then available: OtterlyAI's log experiment, SE Ranking's 300,000-domain analysis and Google's statements. Everything published since - Ahrefs' May 2026 logs, Google's July 2026 guide and Common Crawl's August 2026 analysis - points the same way. None of the four scenarios that would have justified reversing the call have materialized:

  1. A major crawler publicly committing to read the file. Didn't happen. Google said the opposite on the record. OpenAI and Anthropic have not made the commitment despite repeated press cycles.
  2. A citation lift visible at population scale. Didn't happen. The 300,000-domain study found the opposite - including llms.txt in the feature set degraded predictive accuracy.
  3. A meaningful fetch rate from AI-search bots. Didn't happen. About 0.1% of AI bot visits in OtterlyAI's logs; 97% of files with no requests in Ahrefs' May 2026 data.
  4. A spec change that gives search engines a reason to read the file. Didn't happen. Spec v2 (August 2026) added link relations for discovery, the subpath rule and both .md URL forms - all useful for agents, none of them a search signal - and Google's guide still says to ignore the file.

So our policy stays where it has been since March: the audit detects llms.txt when it's present, lists the format issues that would matter if you publish one (no # H1 title, no ## sections, no descriptive paragraph, no markdown links), and stops there. No score impact either way. The check exists because publishing a usable llms.txt is cheap if you choose to do it, and because some teams do use it productively for the developer-tooling case Howard originally described - we want them to be able to verify their file is well-formed without us pretending the file moves their search rank.

What We're Not Saying

A few precise distinctions, because this is the kind of post that attracts straw-man counterarguments.

We are not saying llms.txt is useless. For developer documentation consumed by coding agents (Claude Code, Cursor, Windsurf, Continue), llms.txt is genuinely useful as a curated index the agent can fetch on demand. If your audience is developers building against your API and you maintain llms.txt and llms-full.txt for that purpose, keep doing it.

We are not saying you should remove existing llms.txt files. They are cheap to keep. The cost is in prioritizing the work - putting llms.txt on a content roadmap, paying a consultant to "optimize" it, treating it as a remediation item on an audit. The maintenance hours are real and the SEO outcome is not.

We are not saying no AI system has ever fetched /llms.txt. Some do, occasionally. Anthropic's own documentation discovery probably uses it. Specialized agents pointed at the file by a user will fetch it. The claim we're evaluating is narrower and concrete: does presence of llms.txt change the rate at which your pages get cited in mainstream AI answers (ChatGPT, Claude, Perplexity, Google AI Overviews)? The 2026 evidence says no.

We are not making a claim about 2027. If OpenAI or Anthropic ship a retrieval pipeline that reads llms.txt and the citation evidence flips, we will re-score. The check exists, the file is detected, the data hooks are in place. Re-weighting would be a single config change.

A Practical 5-Minute Decision Tree

If you're trying to decide what to do about llms.txt for your own site, the answer is short:

  1. Are you publishing developer-facing technical documentation that coding agents will consume? If yes - publish llms.txt and llms-full.txt. Keep them current. The use case works.
  2. Do you currently have llms.txt deployed and someone is paying for ongoing optimization of it? Stop the optimization spend. The file is fine to keep; the ongoing tuning is not producing measurable outcomes.
  3. Are you about to add llms.txt as part of an SEO remediation plan? Defer. The Q2 2026 evidence says it won't move citation rates. Spend the hours on the signals that do - original research, firsthand experience patterns, comparison tables, properly structured H2/H3 hierarchies. Our previous posts on original research as an AI citation signal and what AI search engines actually check walk through the alternatives.
  4. Is an audit tool warning you that missing llms.txt is costing your score? Get a different audit tool. Or wait - most of them will catch up to the 2026 evidence in the next quarter.

The honest answer to "should I add llms.txt for AI search visibility in mid-2026?" is no, not for that reason. The honest answer to "is llms.txt a meaningful AI ranking signal?" is not in the data we have, and the data is now substantial enough to act on.

How to Create an llms.txt File

If the decision tree says yes - typically because coding agents read your documentation - the format is small. The spec defines the sections in this order:

  1. An H1 with the site or project name. This is the only required part.
  2. A blockquote with a one- or two-sentence summary.
  3. Optional paragraphs or lists (no headings) with anything an agent needs to know before reading further.
  4. H2 sections with file lists. Each item is a markdown link, [name](url), optionally followed by : and a short note.
  5. An ## Optional section for secondary links an agent can skip when it needs a shorter context. Since spec v2 this is only a convention; tools no longer treat it specially.

A minimal file for a documentation site looks like this:

# Acme API

> REST API for creating and tracking shipments. All endpoints return JSON.

Authentication uses a bearer token; examples assume the v2 API.

## Docs

- [Quickstart](https://acme.example/docs/quickstart.md): first request in five minutes
- [Authentication](https://acme.example/docs/auth.md): tokens, scopes and rotation
- [Shipments API](https://acme.example/docs/shipments.md): create, update, cancel

## Optional

- [Changelog](https://acme.example/docs/changelog.md)

Three details decide whether the file is useful:

  • Link to clean markdown, not HTML. The spec suggests serving a markdown version of each page at the same URL with .md appended (page.html.md) or swapped in (page.md). Pointing the list at HTML pages wrapped in navigation defeats the purpose.
  • Curate, do not dump. A file that lists every URL on the site is a sitemap in markdown. The value is in choosing the pages an agent should read first and saying in one line what each contains.
  • Keep it current. A list pointing at moved or deleted pages is worse than no file. Generate it from the same source as your docs where you can.
  • Serve it as plain text. llmstxt.org, OpenAI and Anthropic serve their files as text/plain. Watch out for single-page apps whose catch-all route answers /llms.txt with the HTML shell: Common Crawl found 40.36% of the responses it collected came back as text/html.

llms-full.txt, which some tools generate alongside, is not part of the llmstxt.org spec. It is a convention from documentation platforms that concatenate every page into one file.

To check the result, follow our llms.txt validation guide: five checks, each with a command, that catch the errors seen most in real files. Our audit runs the format checks too: the llms.txt File check detects the file and offers a format hint, and the Page in llms.txt check notes whether the audited page is listed. A missing file or an unlisted page is reported as information, not a warning, and costs no penalty points - for the reasons laid out above.

What This Means for the Broader "AI SEO" Stack

llms.txt is the second piece of "AI-first" markup to fail the Q2 2026 evidence test in a row. We covered the FAQ rich result deprecation two weeks ago: FAQ schema lost its Google rich result on May 7 and a May 2026 study reported by Search Engine Roundtable found adding schema markup did not improve AI citation rates. llms.txt is now in the same category - a piece of optional markup that the industry sold as a ranking lever and that the data shows isn't one.

The pattern matters more than either individual file. Both were proposed as cheap technical optimizations that would produce a visibility outcome. Both got enthusiastic adoption from teams looking for low-cost wins. Both have now run for long enough to be measured at population scale. Neither produces the outcome it was sold as producing.

The signals that do show up in the post-March-Core-Update analyses - original research, firsthand experience, demonstrably author-produced photos and measurements, properly structured content - all share one property: they cost real human effort to produce. The trade-off is no longer "do the cheap markup work and the algorithm will reward you." It's "do the work the algorithm can detect because the work happened, and the markup is decoration."

Our audit's job is to score against the signals that the evidence supports. When the evidence moves, the weights move. When we find a check that the data no longer justifies, we say so out loud - that's why this post exists, and why the March policy change on FAQ schema looked the same in shape: published evidence, published weight change.

Want to see how your pages score against the May 2026 weights - including the deprecated llms.txt and FAQ-schema signals, and the upgraded Experience and original-research ones? Run a free audit at hybridranking.com.

Sources

  1. The /llms.txt spec, v2 - llmstxt.org
  2. AI optimization guide - Google Search Central
  3. The llms.txt Experiment - OtterlyAI
  4. llms.txt study: May 2026 logs across 137K domains - Ahrefs
  5. A content analysis of llms.txt files from the July 2026 crawl - Common Crawl
  6. LLMs.txt Shows No Clear Effect On AI Citations (300k domains) - Search Engine Journal / SE Ranking
  7. Google says normal SEO works for ranking in AI Overviews and llms.txt won't be used - Search Engine Land (archived)
  8. Google says llms.txt comparable to keywords meta tag - Search Engine Journal
  9. Google: no AI system currently uses llms.txt - Search Engine Roundtable
  10. llms.txt audit - Chrome Lighthouse
  11. Study Says Adding Schema Did Not Improve AI Citations - Search Engine Roundtable
The HybridRanking Advantage

See What AI Search Sees on Your Page

Run a free audit: AI crawler access, structured data, content structure and technical SEO, with a fix for every issue.