Which AI crawlers are there, and what does each do?
Showing 34 of 34 user agents.
| What blocking it does | IP list | |||||
|---|---|---|---|---|---|---|
| GPTBot | OpenAI | Training | Yes | Independent of OAI-SearchBot: you can block GPTBot and still allow ChatGPT search. Effect on other OpenAI answers: not stated. | IP list | October 29, 2026 |
| OAI-SearchBot | OpenAI | Search index | Yes | Site is not shown in ChatGPT search answers, though it can still appear as navigational links. | IP list | October 29, 2026 |
| ChatGPT-User | OpenAI | User fetch | No operator: robots.txt rules may not apply | Not used to decide whether content appears in ChatGPT search. Effect on other answers: not stated. | IP list | October 29, 2026 |
| OAI-AdsBot | OpenAI | Other | Not stated | Not stated | IP list | October 29, 2026 |
| ClaudeBot | Anthropic | Training | Yes | Signals that future material should be excluded from Anthropic's AI training datasets. | IP list | October 29, 2026 |
| Claude-User | Anthropic | User fetch | Yes | Stops Claude retrieving your content for a user's query; may reduce visibility in user-directed search. | IP list | October 29, 2026 |
| Claude-SearchBot | Anthropic | Search index | Yes | Stops indexing for Claude search; may reduce visibility and accuracy in Claude search results. | IP list | October 29, 2026 |
| PerplexityBot | Perplexity | Search index | Yes operator claim; disputed by third-party report | Perplexity recommends allowing it so the site appears in search results. Effect of blocking: not stated. | IP list | October 29, 2026 |
| Perplexity-User | Perplexity | User fetch | No operator: generally ignores robots.txt | Not stated | IP list | October 29, 2026 |
| Google-Extended | Training | Yes robots.txt control token | No effect on Google Search inclusion or ranking. Effect on Gemini answers beyond training and grounding: not stated. | IP list | October 29, 2026 | |
| Googlebot | Search index | Yes | Affects Google Search, Images, Video and News. Do not block it to opt out of AI training. | IP list | October 29, 2026 | |
| Google-CloudVertexBot | Other | Yes robots.txt token documented | Google says it has no effect on Google Search or other products. | IP list | October 29, 2026 | |
| Google-Agent | User fetch | No Google: user-triggered fetchers generally ignore robots.txt | Not stated | IP list | October 29, 2026 | |
| Google-GeminiNotebook | User fetch | No Google: user-triggered fetchers generally ignore robots.txt | Not stated | IP list | October 29, 2026 | |
| Bytespider | ByteDance | Not stated | Unverified | Not stated | Not stated | October 29, 2026 |
| CCBot | Common Crawl | Other | Yes | Not stated | IP list | October 29, 2026 |
| Amazonbot | Amazon | Training | Yes | Not stated | IP list | October 29, 2026 |
| Amzn-SearchBot | Amazon | Search index | Yes | Allowing it makes content eligible to appear in search experiences such as Alexa. | IP list | October 29, 2026 |
| Amzn-User | Amazon | User fetch | Partial operator: may not follow all robots.txt directives | Not stated | IP list | October 29, 2026 |
| Applebot-Extended | Apple | Training | Yes | Pages can still be included in Apple search results. | IP list | October 29, 2026 |
| Applebot | Apple | Search index | Yes | Allowing it lets content appear in Apple search results in Spotlight, Siri and Safari. | IP list | October 29, 2026 |
| Meta-ExternalAgent | Meta | Training | Yes operator gives robots.txt instructions | Not stated | Not stated | October 29, 2026 |
| Meta-ExternalFetcher | Meta | User fetch | No operator: may bypass robots.txt | Not stated | Not stated | October 29, 2026 |
| Meta-WebIndexer | Meta | Search index | Not stated robots.txt control implied | Allowing it helps Meta cite and link to your content in Meta AI. Effect of blocking: not stated. | Not stated | October 29, 2026 |
| DuckAssistBot | DuckDuckGo | User fetch | Yes | No effect on DuckDuckGo search rankings. Removes the site as a source for AI-assisted answers. | IP list | October 29, 2026 |
| cohere-ai | Cohere | Not stated | Not stated | Not stated | Not stated | October 29, 2026 |
| Diffbot | Diffbot | Search index | Yes | Crawling lets sites be discovered and cited from the Diffbot Knowledge Graph. Effect of blocking: not stated. | Not stated | October 29, 2026 |
| Diffbot-User | Diffbot | User fetch | Not stated | Not stated | Not stated | October 29, 2026 |
| YouBot | You.com | Search index | Yes | Not stated | See crawler page | October 29, 2026 |
| Timpibot | Timpi | Not stated | Unverified | Not stated | Not stated | October 29, 2026 |
| omgili | Webz.io | Other | Yes operator claim | Not stated | Not stated | October 29, 2026 |
| MistralAI-User | Mistral AI | User fetch | Not stated | Not stated | IP list | October 29, 2026 |
| MistralAI-Index | Mistral AI | Search index | Not stated | Indexes content for Mistral search in Vibe. Effect of blocking: not stated. | IP list | October 29, 2026 |
| MistralAI-Training | Mistral AI | Training | Yes operator: webmasters can disallow it in robots.txt | Not used for search indexing or to answer live user queries in Vibe. | Not stated | October 29, 2026 |
Select a crawler for its exact user agent string, copy-paste robots.txt lines, IP range link and sources. Counts and dates above come from the same file as the table.
How do I block an AI crawler in robots.txt?
Add a group to the robots.txt file at the root of your site. The group names the crawler, then says what it may not fetch. To keep GPTBot out of the whole site, the group is "User-agent: GPTBot" followed by "Disallow: /".
Name each crawler. A wildcard group would also block Googlebot and Bingbot. Our robots.txt generator for AI crawlers writes the file for you. Our guide on how to block or allow ChatGPT in robots.txt covers the OpenAI names in detail.
Which AI crawlers should I block, and which should I allow?
Sort them by what they do. The cautions below apply to specific rows, and each crawler page has the operator's own wording.
- Training: 7 rows. Blocking a training crawler is a choice about future model training. OpenAI says GPTBot and OAI-SearchBot settings are independent, so a site can disallow GPTBot and still allow OAI-SearchBot (OpenAI, read October 29, 2026).
- Search index: 10 rows. These build the index a search or AI answer product draws on. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers" (OpenAI, read October 29, 2026).
- User fetch: 10 rows. These open one page because a person asked an AI tool to. Several operators say robots.txt may not apply to them: OpenAI for ChatGPT-User, Perplexity for Perplexity-User, Meta for Meta-ExternalFetcher, and Google for its user-triggered fetchers.
- Control tokens are not crawlers. Google-Extended and Applebot-Extended only control how pages crawled by Googlebot and Applebot may be used. Blocking them does not remove a site from Google Search or Apple search.
- Unverified: Bytespider and Timpibot have no operator documentation we could find, and Cohere says it does not crawl for generative AI training at this time. Treat any compliance claim about them as unconfirmed.
Which tool writes the robots.txt for me?
Use the robots.txt generator for AI crawlers: tick the bots you want in or out and copy the file. Blocking is only half the job. The llms.txt generator writes a short map of your site for the AI tools you do let in. If you want articles on your site that those tools can quote, see Paperboy pricing. Your first article is free.
How is this list built, and how current is it?
We opened each operator's crawler documentation and copied the user agent strings, robots.txt tokens, IP range links and blocking wording from it. The read date is on every row and every source link. User agent version numbers (GPTBot/1.4, for example) change, so the date matters.
We did not download or parse the IP range files, apart from checking that Anthropic's list is live. Robots.txt is a request, not access control: RFC 9309 says its rules "are not a form of access authorization" (RFC 9309, read October 29, 2026). A crawler that ignores it can only be stopped at your server or CDN.
Can I download the AI crawler dataset?
Yes. It is free to use under CC BY 4.0, which means you may copy and reuse it with credit.
- ai-crawlers.json (34 rows, the canonical file)
- ai-crawlers.csv (same rows; tokens and sources joined with " | ")
- All Paperboy open data, including the October 26, 2026 robots.txt snapshot of 55 sites (how it was made)
If you use it, please cite it:
Cite as: Paperboy AI Crawler Dataset, paperboyseo.com/ai-crawlers, updated October 29, 2026
Found a wrong or missing row? Tell us on the contact page and cite the operator page that shows it.
Questions about AI crawlers and robots.txt
Does blocking Google-Extended remove my site from Google Search?
No. Google-Extended is a robots.txt control token, not a crawler. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search" (Google Search Central, read October 29, 2026). Blocking Googlebot is what removes a site from Search.
Does blocking Applebot-Extended remove my site from Apple search?
No. Apple says "Webpages that disallow Applebot-Extended can still be included in search results" (Apple Support, read October 29, 2026). The token only controls whether Apple may use pages Applebot already crawled to train its foundation models.
Will robots.txt stop ChatGPT-User or Perplexity-User?
Not reliably. OpenAI says that because ChatGPT-User actions are started by a user, "robots.txt rules may not apply", and Perplexity says Perplexity-User generally ignores robots.txt (both read October 29, 2026). Stopping them takes a server or CDN rule.
Does PerplexityBot obey robots.txt?
Perplexity says you can control it with robots.txt. In August 2025 Cloudflare alleged that Perplexity also used undeclared crawlers with generic browser user agents on sites that disallowed its declared bots, and Perplexity disputed that. We list the operator's claim and the dispute side by side and make no ruling.
Why are some rows marked unverified?
For Bytespider and Timpibot we found no operator documentation, and Cohere's own page says it does not crawl for generative AI training at this time. We say "not stated" or "unverified" in those rows instead of repeating third-party claims.
Next: give AI tools something to quote
Paperboy writes articles that answer the questions your buyers ask, with every fact linked to its source, FAQ and Article structured data, and publishes them to your site every week. Your first article is free.
How do I cite this page?
<a href="https://paperboyseo.com/ai-crawlers">Paperboy AI crawler directory</a> (Paperboy)License: CC BY 4.0. Use it, share it and adapt it, and credit Paperboy with a link to this page.