LLM SEO: How Large Language Models Find, Learn From and Cite Your Site

When someone asks ChatGPT, Claude or Gemini about your product, the answer can come from two very different places. It can come from what the model learned during training - knowledge that, as the authors of the original retrieval-augmented generation paper put it, language models "store ... in their parameters". Or it can come from pages the system fetches at the moment of the question and quotes back, with links.

"LLM SEO" - also called LLM optimization or LLMO - is the work of being well represented in both. Most advice mixes the two, which leads to expensive mistakes: blocking a training crawler in the belief that it hides a site from AI answers, or allowing everything and expecting to be cited. This guide separates the two channels, names the crawler behind each, and shows which parts you can actually influence.

A note on the term. In machine learning, "LLM optimization" usually means making models run faster and cheaper - NVIDIA and Hugging Face use it for inference optimization. This article is about the marketing meaning, which Wikipedia lists as another name for generative engine optimization.

Key Takeaways

• There are two channels. Training shapes what a model knows by default; retrieval decides what it reads and cites for a specific question.
• Each channel has its own crawler. GPTBot, ClaudeBot and Google-Extended govern training; OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot govern the search indexes AI answers draw from.
• Blocking training does not block answers. OpenAI's documentation gives the example of allowing OAI-SearchBot to appear in search results while disallowing GPTBot.
• Retrieval works on pieces of pages. Microsoft describes AI search as parsing content into pieces and selecting "which pieces of content earn a place in the final answer".
• Models read more than they cite. OpenAI's web search tool returns a full list of sources consulted, which can be longer than the list of citations.

Two Ways an LLM Learns About Your Site

Training. During training, a model absorbs patterns and facts from a large text corpus, including crawled web pages. What it learns is fixed until the next model version. You can decide whether your content is used - by allowing or blocking training crawlers - but there is no way to request that a model learns specific facts about you.

Retrieval. When a question needs current or specific information, the system searches, fetches pages and grounds its answer in them. Google's AI optimization guide calls this retrieval-augmented generation, "also known as grounding", built on its core ranking systems and Search index. This is the channel that produces citations and clicks, and the one SEO work affects most directly.

Action: decide on each channel separately: whether your content may be used for training, and whether your pages should be findable for live answers.

Which Crawler Controls Which Channel

Two details change how you read this table. Google-Extended is not a crawler; it tells Google whether content Googlebot already fetched may be used for Gemini training and grounding, and Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". And user-triggered fetchers often ignore robots.txt: OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules". Our AI crawlers list covers 31 agents in detail, and our guides to GPTBot and ClaudeBot go bot by bot.

Action: check that every search-index crawler in the middle column can reach your important pages, whatever you decide about training.

What Happens When an AI Answers a Question

The retrieval channel is a pipeline, and each step is a place where a page can drop out:

1. The question is expanded. Google describes AI Overviews and AI Mode as using "query fan-out" - issuing multiple related searches across subtopics and data sources.
2. The system searches its index. OpenAI's web search tool exposes this as search, then open_page and find_in_page actions; a reasoning model can "decide whether to keep searching".
3. Pages are split and filtered. Microsoft's Bing grounding documentation says the search returns "relevant chunks", and the model "may also decide which pieces of tool outputs are used".
4. The answer cites a subset. OpenAI's tool lists all sources consulted separately from the ones cited, so being read is not the same as being credited.

Microsoft summarized the shift in its October 2025 guidance: ranking is now "less about ordering entire pages and more about which pieces of content earn a place in the final answer".

Action: make sure each important page answers its main question in a self-contained passage near the top; our front-loading check and answer passages check test this.

What You Can Influence in Each Channel

In training, very little. You can allow or block training crawlers. You cannot request inclusion, correct a model directly or choose what it remembers. Changes appear only in future model versions.

In retrieval, most of SEO applies. Pages must be crawlable by the search crawlers, indexed, eligible for snippets and readable without JavaScript - in Vercel and MERJ's 2024 measurements, none of the major AI crawlers executed JavaScript. Beyond that, passage quality matters: the GEO study found that adding quotations, statistics and cited sources raised a page's visibility in AI answers, while keyword stuffing lowered it. Google's position is that none of this needs special markup or files: "You don't need to write in a specific way just for generative AI search."

Action: put your effort where it pays back fastest - crawler access, server-rendered content and clear, sourced passages - rather than trying to steer training.

A Common LLM SEO Mistake: Blocking the Wrong Channel

A frequent robots.txt problem is blocking the wrong channel. A User-agent: * group with Disallow: /, added to stop "AI training", also blocks every search crawler without its own group - OAI-SearchBot, Claude-SearchBot, PerplexityBot - and removes the site from the answers it wanted to protect. The opposite also happens: a site blocks GPTBot alone and assumes it has left ChatGPT, while OAI-SearchBot keeps indexing it.

The fix is to write training and search as separate groups:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

Action: open your robots.txt and confirm the search crawlers have an explicit Allow or at least are not caught by a wildcard block; our AI bot access check does this for 15 agents.

An LLM SEO Checklist

1. Decide on training: allow or block GPTBot, ClaudeBot, Google-Extended and other training agents.
2. Allow the search-index crawlers for every AI answer surface you care about.
3. Serve important content in the initial HTML, not only after JavaScript runs (JavaScript content gap check).
4. Answer each page's main question in the first passage, with a number or source where you have one.
5. Cover the follow-up questions a fan-out search would ask on the same page.
6. Earn mentions on third-party sites that AI answers already cite - see GEO vs SEO for the data.
7. Measure citations and AI referrals over weeks, not single prompts - see how to measure AI visibility.

For the writing side, our answer engine optimization guide turns these principles into rules for individual passages. To check crawler access, rendering and passage structure on your own pages, run a free audit.

Sources

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks - Lewis et al., NeurIPS 2020
2. AI optimization guide - Google Search Central
3. AI features and your website - Google Search Central
4. Google's common crawlers (Google-Extended) - Google Search Central
5. Overview of OpenAI Crawlers - OpenAI
6. Web search tool - OpenAI API documentation
7. Perplexity Crawlers - Perplexity
8. Grounding with Bing Search - Microsoft Learn
9. Optimizing your content for inclusion in AI search answers - Microsoft Advertising, October 2025
10. GEO: Generative Engine Optimization - Aggarwal et al., KDD 2024
11. The rise of the AI crawler - Vercel and MERJ, December 2024
12. Mastering LLM Techniques: Inference Optimization - NVIDIA
13. Optimizing inference - Hugging Face
14. Generative engine optimization - Wikipedia