Technical SEO
Important
AI Bot Access Rules
search What we check
Whether robots.txt has explicit rules for each major AI bot, including GPTBot, PerplexityBot, ClaudeBot, and others.
build How to fix
- 1 Add explicit User-agent rules for each AI bot in robots.txt.
- 2 To allow: User-agent: GPTBot / Allow: /.
- 3 To block training crawlers while allowing search bots, set rules per bot.
- 4 Review access for all bots: GPTBot, Google-Extended, PerplexityBot, ClaudeBot, OAI-SearchBot, ChatGPT-User, Bingbot, Googlebot.
menu_book Sources
Does your site pass this check?
This is 1 of 98 checks we run. Find out in under 30 seconds.