Quick Takeaway
Control AI crawlers with robots.txt by naming each crawler's user agent. Vendors run separate bots for training (such as GPTBot and ClaudeBot), for search results (such as OAI-SearchBot and PerplexityBot) and for user-requested visits. Most businesses should allow the search bots so they can be cited, and decide on training bots separately. llms.txt is an unofficial proposal with no confirmed ranking benefit.
Key takeaways
- AI companies run different crawlers for training, search and user-requested fetches, and you can treat them differently.
- Blocking a training crawler does not remove you from that vendor's search or answers. Blocking a search crawler can.
- Google-Extended controls Gemini training and grounding only. It does not affect Google Search inclusion or ranking.
- Google says you need no special file or markup to appear in AI Overviews or AI Mode. Standard SEO applies.
- robots.txt is a request, not a lock. Use your server logs to see who really visits, and a firewall to enforce.
More of your visitors now arrive through AI tools, and behind each of those tools is a crawler visiting your site. Some collect content to train models. Some build the index that powers AI search. Some fetch a page because a user just asked about it. Treating them all the same means either hiding from the tools that could send you customers or handing content to ones you would rather opt out of.
This guide explains what each crawler is for, how to set rules, and where the hype around llms.txt outruns the evidence. It complements our post on how to get cited by AI search, which covers the content side. Crawler names and behavior change, so check each vendor's documentation (linked below) before you rely on a rule.
The main AI crawlers and what they do
| Company | User agent | Purpose | robots.txt |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Surfaces sites in ChatGPT search results | Can be controlled. Opting out means not appearing in ChatGPT search answers |
| OpenAI | GPTBot | Crawls content that may be used to train generative AI models | Can be controlled. Disallowing signals your content should not be used for training |
| OpenAI | ChatGPT-User | Visits pages when a user asks ChatGPT to | OpenAI says robots.txt rules may not apply, because a user initiated it |
| Anthropic | ClaudeBot | Collects public web content that may contribute to training | Anthropic states its bots honour robots.txt |
| Anthropic | Claude-SearchBot | Improves the quality of search results in Claude | Anthropic states its bots honour robots.txt |
| Anthropic | Claude-User | Visits pages when a person asks Claude a question | Anthropic states its bots honour robots.txt |
| Perplexity | PerplexityBot | Surfaces and links sites in Perplexity results. Not used for foundation-model training | Respects robots.txt |
| Perplexity | Perplexity-User | Visits a page when a user asks a question | Generally ignores robots.txt, because a user requested the fetch |
| Googlebot | Crawls for Google Search, which includes AI features built into Search | Respects robots.txt | |
| Google-Extended | A robots.txt token controlling use of crawled content for training future Gemini models and grounding in Gemini Apps | Token only. Does not affect Search inclusion or ranking |
Training and search are different switches
Disallowing a training crawler such as GPTBot or ClaudeBot tells that company not to use your content for training. It does not, on its own, remove you from that company's search product, which uses a separate crawler. To stay out of ChatGPT search results you would have to disallow OAI-SearchBot as well, which most businesses do not want.
What robots.txt can and cannot do
- robots.txt
- A plain-text file at the root of your site that tells crawlers which paths they may fetch. It follows the Robots Exclusion Protocol (RFC 9309) and works by convention: well-behaved crawlers honour it, but nothing forces them to.
- It requests compliance. Reputable vendors follow it, but scrapers with no reputation to protect ignore it.
- User-requested fetchers (such as ChatGPT-User and Perplexity-User) may not follow it, because a person asked for the page.
- It controls crawling, not appearance. To limit how Google shows your content in search and AI features, use noindex, nosnippet, data-nosnippet or max-snippet.
- Blocking a crawler by IP address can backfire: Anthropic notes that doing so may stop its crawler reading your robots.txt at all.
Sample robots.txt rules
A common approach for a business that wants to be found is to allow search and retrieval crawlers and make a separate choice about training. The example below allows AI search bots and opts out of training.
# Allow AI search and retrieval crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Opt out of model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Everyone else
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlTest before you publish
A stray "Disallow: /" under the wrong user agent can hide your site from search. Never put it under "User-agent: *" unless you mean to block everything. Check the file with a robots.txt tester and confirm Googlebot can still fetch your key pages.
Should you block AI crawlers?
| Your situation | Sensible starting point | Why |
|---|---|---|
| Business that wants leads and enquiries | Allow search and retrieval crawlers | Being cited and linked in AI answers can send qualified visitors |
| Publisher whose content is the product | Allow search crawlers, consider blocking training crawlers | Protects the value of original content while staying visible |
| Members-only or private content | Block, and protect it behind a login | robots.txt is not security. Do not rely on it to hide private pages |
| Unsure | Allow, then review your server logs | You can tighten the rules later once you see who is visiting |
What about llms.txt?
llms.txt is a proposal to put a Markdown file at /llms.txt that summarizes your site and links to your most useful pages in a format language models can read easily. It is not an official standard, and adoption by AI providers is unclear. Google is explicit about its own products: you do not need to create new machine-readable files, AI text files or special markup to appear in AI Overviews or AI Mode, and no special schema.org markup is required. We have not found documentation from OpenAI, Anthropic or Perplexity confirming that they use llms.txt as a ranking or citation signal.
Publishing one is cheap and harmless, so it is fine if you want to. Just do not expect it to move rankings, and do not let it distract from work that demonstrably matters.
What matters more for AI visibility
- Crawlable HTML. Your main content is in the server-rendered page, not hidden behind scripts or logins. See our technical SEO checklist.
- Clear, direct answers. Each page answers a specific question near the top, then adds detail.
- Consistent facts. Your name, services, prices and contact details match everywhere they appear.
- Structured data. Valid Organization, Article and other markup that matches what is visible.
- Real expertise. Named authors, original examples and sources that a system can trust.
- A healthy site. Fast, mobile-friendly, correct status codes, and no accidental noindex.
How to see who is crawling you
Step 1: Check your server or CDN logs
Search for the user agents above to see which AI crawlers visit and how often.
Step 2: Look at referral traffic
In your analytics, watch for visits arriving from AI assistants and search tools, and note which pages they land on.
Step 3: Ask the assistants directly
Put your buyers' real questions to several AI tools and check whether your business is mentioned and described correctly.
Step 4: Review quarterly
Crawler names, policies and products change. Recheck the vendor documentation and your rules a few times a year.
Our SEO service includes this kind of crawl and visibility review, and you can book a free strategy call if you would like a second opinion on your rules and your site.
AI crawlers: common questions
Once AI crawlers can reach your site, give them something worth citing: our guide to writing blog posts that rank in Google and AI search explains how. To see how ChatGPT, Google AI Overviews and Copilot decide which businesses to name, read how to get your business recommended by ChatGPT.
How do I block GPTBot?
Add a group to your robots.txt with "User-agent: GPTBot" followed by "Disallow: /". This signals that your content should not be used to train OpenAI's generative AI models. It does not affect OAI-SearchBot, which is a separate crawler for ChatGPT search.
Does blocking AI crawlers hurt my Google rankings?
Blocking AI training crawlers does not affect Google Search. Google states that Google-Extended does not impact a site's inclusion in Google Search and is not a ranking signal. Blocking Googlebot itself is different and can remove you from Google's results.
Do AI crawlers always obey robots.txt?
Reputable vendors say their crawlers do, with one notable exception: fetchers triggered by a user's request, such as ChatGPT-User and Perplexity-User, may not follow robots.txt. Unknown scrapers may ignore it entirely, so use a firewall if you need enforcement.
Do I need an llms.txt file?
No. It is an unofficial proposal, Google says no special file is needed to appear in its AI features, and we have not found confirmation from other major providers that they use it. Publish one if you like, but focus on crawlable content and clear answers.
How do I get my business mentioned in ChatGPT or Perplexity?
Allow their search crawlers, keep your content crawlable and factual, answer specific questions directly, use accurate structured data, and build genuine expertise and references. There is no guaranteed method, and results vary by query.