AI companies send different bots to your website for different reasons. Some fetch pages so your business can appear, with a link, in AI search answers. Others collect text to train future AI models. A third kind visits a page only because a user asked an assistant to look at it.
Treating them all the same is the most common mistake we see. Block the wrong one and your business can disappear from AI search results. Allow everything without thinking and you may be fine, but you should know what you're agreeing to.
The short answer: if you want your business to be found in AI answers, allow the search crawlers and the user-triggered fetchers. Whether to allow training crawlers is a separate business decision, and blocking them doesn't remove you from search. For Google's AI Overviews and AI Mode, what matters is normal Googlebot access, not a special AI bot.
Checked against each company's own documentation on 25 September 2026. Bot names and rules change, so follow the links before you edit your robots.txt.
The Three Kinds of AI Bots
1. Search and retrieval crawlers
These visit your pages so an AI product can find them, show them in results and link to them. Blocking them is the closest thing to "opting out of AI search" for that product.
2. Training crawlers (and training controls)
These collect content that may be used to train future AI models. Some companies use a separate robots.txt name that doesn't crawl at all, but tells them whether content their normal crawler already collected may be used for training.
3. User-triggered fetchers
These visit a page only when a person asks an assistant to open or check it. Because a person started the request, several companies say these fetchers may not follow robots.txt rules.
The Bots, Company by Company
OpenAI (ChatGPT)
- OAI-SearchBot: search. Used to "surface websites in search results in ChatGPT's search features". According to OpenAI, sites that block it "will not be shown in ChatGPT search answers", although they can still appear as navigational links.
- GPTBot: training. OpenAI says disallowing GPTBot "indicates a site's content should not be used in training".
- ChatGPT-User: user-triggered. Visits pages when a user asks ChatGPT to; "robots.txt rules may not apply", and it isn't used to decide search eligibility.
Source: OpenAI's crawler documentation
Anthropic (Claude)
- Claude-SearchBot: search. Improves the quality of Claude's search results; blocking it may reduce your visibility there.
- ClaudeBot: training. Blocking it signals that your content should be excluded from training data.
- Claude-User: user-triggered. Fetches pages when a Claude user asks a question; blocking it stops Claude retrieving your pages for those requests.
Source: Anthropic's crawler documentation
Perplexity
- PerplexityBot: search. Used to "surface and link websites in search results on Perplexity"; Perplexity recommends allowing it.
- Perplexity-User: user-triggered. Perplexity says this fetcher "generally ignores robots.txt rules" because a user started the request.
Source: Perplexity's crawler documentation
- Googlebot: the normal Search crawler, and the one that matters for AI Overviews and AI Mode. Google says robots.txt rules for Googlebot are "the control for site owners to manage access to how their sites are crawled for Search", and that the usual snippet and indexing controls (
nosnippet,max-snippet,noindex) apply to its AI features too (Google's AI features guidance). - Google-Extended: a training and grounding control for Gemini models, Gemini Apps and Vertex AI. Google states that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" (Google's crawler overview).
Apple
- Applebot: crawls for search features in Spotlight, Siri and Safari.
- Applebot-Extended: a training control only. Apple says it "does not crawl webpages", and pages that disallow it "can still be included in search results".
Source: Apple's Applebot documentation
Microsoft and Common Crawl
- Bingbot crawls for Bing search, which Microsoft's Copilot also draws on for web answers. Blocking Bingbot affects both.
- CCBot builds Common Crawl, a public web archive that has been used to train many AI models. It doesn't power any AI search product directly.
Two Example robots.txt Files
Allow everything (what most small businesses want, and what we do on nhweb.eu):
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
Stay in AI search, but opt out of AI training:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
The second file keeps the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot) and the user fetchers allowed, because none of them are named in a Disallow rule.
What robots.txt Can't Do
- It's a request, not a lock. Well-behaved bots follow it; it doesn't protect private content. Use passwords for that.
- User-triggered fetchers may not follow it, as OpenAI and Perplexity both say.
- Your host or CDN may block bots anyway. Some hosting and CDN providers offer a one-click "block AI bots" setting or bot-protection rules that act before robots.txt is ever read. If your robots.txt allows AI search crawlers but your firewall blocks them, the firewall wins.
- Allowing crawlers doesn't guarantee a mention. Access only makes you eligible. Whether an assistant names your business depends on how clear, consistent and well corroborated your information is.
How to Check Your Own Site in Five Minutes
- Open
yourdomain.com/robots.txtin a browser and look forDisallow: /under any of the bot names above. - Check whether your CMS, hosting or CDN has an "AI bot" or "bot protection" setting switched on.
- Make sure nothing important is hidden behind a login, loaded only by JavaScript, or available only as an image or PDF.
- Search for your business in the assistants you care about and note which sources they cite. Our 15-minute AI self-check walks you through it.
Frequently Asked Questions
Q: If I block GPTBot, will my business disappear from ChatGPT search? A: According to OpenAI's documentation, no. GPTBot is for training; ChatGPT search uses OAI-SearchBot. They are separate controls.
Q: Does blocking Google-Extended remove me from AI Overviews? A: No. Google says Google-Extended doesn't affect inclusion in Google Search. AI Overviews and AI Mode follow your normal Googlebot and snippet settings.
Q: Should I add an "llms.txt" file? A: It does no harm, but Google's guidance says no special files or markup are needed for its AI features. Clear pages that can be crawled and indexed matter far more.
Q: Is it bad to let AI companies train on my website? A: It's a business choice, not an SEO requirement. Many small businesses are happy for public marketing pages to be used; others prefer to opt out. Either way, keep the search crawlers allowed if you want to be found.
Access Is Step One
Letting the right bots in makes your business eligible to appear in AI answers. Being named depends on much more. Read how our AI search optimization works, or get a free AI visibility check to see what AI tools say about your business today.