Yes. robots.txt can allow or block most AI crawlers by user-agent name. For ChatGPT search, the name is OAI-SearchBot. GPTBot is OpenAI's training crawler. ChatGPT-User fetches pages when a person asks, and OpenAI says robots.txt rules may not apply to it (OpenAI, read October 23, 2026).
To see what other sites actually do, our dated robots.txt snapshot of 55 well-known sites shows who blocks GPTBot, ClaudeBot and PerplexityBot. For every AI user agent we could verify, with its exact robots.txt lines and sources, see the AI crawlers list.
Which crawler does what
Each fact in the table comes from the owner's own page, read October 23, 2026. "Not listed" means we did not see it on the page.
| Crawler | Owner | What it does | robots.txt token | Source |
|---|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used in training generative AI foundation models | GPTBot | OpenAI |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT's search features. Opted-out sites will not be shown in ChatGPT search answers | OAI-SearchBot | OpenAI |
| ChatGPT-User | OpenAI | Visits a page for certain user actions. OpenAI says robots.txt rules may not apply | ChatGPT-User | OpenAI |
| ClaudeBot | Anthropic | Collects web content that could contribute to model training. Blocking it excludes future material from training datasets | ClaudeBot | Anthropic |
| Claude-SearchBot | Anthropic | Improves the quality of search results. Blocking it may reduce visibility in Claude search results | Claude-SearchBot | Anthropic |
| Claude-User | Anthropic | Accesses sites when a person asks Claude a question. Blocking it may reduce visibility for user-directed search | Claude-User | Anthropic |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search results. Not used to crawl content for AI foundation models | PerplexityBot | Perplexity |
| Perplexity-User | Perplexity | Visits a page when a user asks a question. Perplexity says it generally ignores robots.txt | Perplexity-User | Perplexity |
| Google-Extended | Controls whether crawled content may be used to train future Gemini models. It does not affect inclusion in Google Search | Google-Extended | ||
| Googlebot | Crawls for Google's search indexes. Its rules affect Google Search and its features | Googlebot | ||
| Applebot-Extended | Apple | Lets you opt out of content being used to train Apple's foundation models. It does not crawl, and disallowed pages can still appear in search | Applebot-Extended | Apple |
| CCBot | Common Crawl | Builds an open repository of web crawl data. Common Crawl documents blocking it with a robots.txt rule | CCBot | Common Crawl |
The pattern: each company has a training crawler and a search crawler, and they are separate tokens. You can block one and allow the other. The user-triggered fetchers (ChatGPT-User, Perplexity-User) are the exception. OpenAI and Perplexity both say robots.txt may not or generally does not apply to them, and Google says the same about its own user-triggered fetchers (Google). Anthropic, by contrast, lists Claude-User as something you can disable in robots.txt (Anthropic).
What Google says about AI Overviews
Google says AI is built into Search, so "robots.txt directives for Googlebot is the control for site owners to manage access." To limit what Search shows from your pages, it points to nosnippet, data-nosnippet, max-snippet and noindex. It also says there are no additional requirements to appear in AI Overviews or AI Mode (Google, read October 23, 2026). So blocking Google-Extended is not how you leave AI Overviews. It is about Gemini training.
Snippet 1: allow all AI search crawlers
Use this if you want to be readable by AI search tools. These groups name the search and user-action bots. Lines for a bot that is not listed fall under your * group.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Googlebot
Allow: /
Allow rules are not required. A bot with no rule is not blocked. Naming them makes your intent explicit, which helps the next person who edits the file.
Snippet 2: block training, allow search
Use this if you do not want your content used for model training but still want to appear in AI search answers.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot
Allow: /
Each group applies only to the bot it names. Disallowing GPTBot does not disallow OAI-SearchBot, because OpenAI treats them as separate settings. OpenAI says a robots.txt update can take about 24 hours to adjust for search (OpenAI, read October 23, 2026).
Test it
- Open
yourdomain.com/robots.txtin a browser and read each group. One block per bot, no typos in the names. - Check your CDN or firewall. A bot rule there can block a crawler even when robots.txt allows it. This is the most common reason an allowed bot never shows up.
- Make sure the page text is in the HTML. If a page only shows its content after scripts run, a crawler may see little.
- Run the checker. It shows whether Google, Bing, ChatGPT search, Perplexity and Claude can read your site, plus sitemap and page basics. It is free and needs no sign-up. Check your site free.
If you want more on the individual names, see Should you block GPTBot? and Block ChatGPT-User in robots.txt?. For the page side of it, see How to get cited by ChatGPT and Perplexity.
What robots.txt cannot do
- It is a request, not a lock. It tells well-behaved crawlers what you prefer. It does not stop a person or a bot that ignores it. For a page that must stay private, use a login or server controls.
- It does not control user-triggered fetches. See the table: OpenAI and Perplexity say their user-action fetchers may or generally do ignore it.
- It is not retroactive. Our reading of the OpenAI and Anthropic pages is that a rule applies to future crawling. Neither page says it removes content from models already trained, so do not assume it does.
- It does not promise citations. Allowing OAI-SearchBot or PerplexityBot makes a page eligible to be read. It does not mean ChatGPT or Perplexity will show or cite it, and nobody, including us, can promise that.
To see how common blocking is, and what AI search traffic looks like, read our AI search statistics for 2026, which includes our own count of who blocks AI crawlers.
Our view
This is our judgment, not a sourced fact. For most small business sites, treat it as two questions. Training: do you mind your content being used to train models? Search: do you want to be findable in AI answers? Answer them separately and use the matching snippet. Then check the CDN, which is where most surprises live.
Paperboy is an AI SEO article writer and publisher by Hive Asset Group, LLC. If you want articles on your site that these crawlers can read once you let them in, see how Paperboy works.
Frequently asked questions
Does robots.txt block ChatGPT?
Partly. OpenAI says GPTBot and OAI-SearchBot follow robots.txt, so you can block training or ChatGPT search with separate rules. For ChatGPT-User, OpenAI says robots.txt rules may not apply because a user starts the action (OpenAI, read October 23, 2026).
Which robots.txt line lets ChatGPT search read my site?
Add a group with User-agent: OAI-SearchBot and Allow: /. OpenAI says this crawler is used to surface websites in ChatGPT's search features, and sites opted out will not be shown in ChatGPT search answers. A site with no rule is not blocked (OpenAI, read October 23, 2026).
Can I block AI training but stay in AI search?
Yes. Disallow the training tokens and allow the search ones. OpenAI says disallowing GPTBot means content should not be used in training, and that OAI-SearchBot is a separate setting. Anthropic and Perplexity also separate their training and search bots (Anthropic, Perplexity).
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search. For AI Overviews, Google says Googlebot rules are the control, and snippet controls such as nosnippet and max-snippet limit what Search shows (Google, Google AI features, read October 23, 2026).
How long does a robots.txt change take?
For ChatGPT search, OpenAI says it can take about 24 hours from a site's robots.txt update for its systems to adjust. We did not see a stated delay for the other crawlers on their official pages, so check again after a day or two (OpenAI, read October 23, 2026).