Robots.txt Generator for AI Crawlers (Free)

Robots.txt generator for AI crawlers

Pick which AI crawlers may read your site. Start from a preset (allow AI search and block training, block all AI, or allow all), then change any crawler. The tool writes the robots.txt lines as you click. Copy them or download the file. It all runs in your browser, so nothing you type is sent to us.

Crawler	What it does	Rule
GPTBot OpenAI	Training. Collects pages that may be used to train OpenAI's generative AI models.	Rule for GPTBot
OAI-SearchBot OpenAI	AI search. Surfaces sites in ChatGPT search results.	Rule for OAI-SearchBot
ChatGPT-User OpenAI	User fetch. Opens a page when someone asks ChatGPT to read it. OpenAI says robots.txt rules may not apply. May ignore robots.txt.	Rule for ChatGPT-User
ClaudeBot Anthropic	Training. Collects pages that may be used to train Claude models.	Rule for ClaudeBot
Claude-User Anthropic	User fetch. Opens a page when someone asks Claude to read it.	Rule for Claude-User
Claude-SearchBot Anthropic	AI search. Indexes pages to improve the quality of Claude search results.	Rule for Claude-SearchBot
PerplexityBot Perplexity	AI search. Indexes pages so they can appear in Perplexity answers.	Rule for PerplexityBot
Perplexity-User Perplexity	User fetch. Opens a page when someone asks Perplexity to read it. Perplexity says it generally ignores robots.txt. May ignore robots.txt.	Rule for Perplexity-User
Google-Extended Google	Training. A control token, not a crawler: sets whether Google may use your pages for Gemini training and grounding. It does not affect Google Search.	Rule for Google-Extended
Bytespider ByteDance	Purpose unverified. ByteDance's crawler. ByteDance publishes no documentation we could find, so its purpose is unverified.	Rule for Bytespider
CCBot Common Crawl	Data collection. Builds the Common Crawl open web archive, which anyone can download.	Rule for CCBot
Amazonbot Amazon	Training. Amazon's crawler, used to improve its products and services. It may be used to train Amazon's AI models.	Rule for Amazonbot
Applebot-Extended Apple	Training. A control token: sets whether Apple may use pages Applebot crawled to train its foundation models. Search inclusion is unaffected.	Rule for Applebot-Extended
Meta-ExternalAgent Meta	Training. Crawls to train Meta's AI models and to index content directly.	Rule for Meta-ExternalAgent
Meta-ExternalFetcher Meta	User fetch. Opens a page when a user action in a Meta AI product needs it. Meta says it may bypass robots.txt. May ignore robots.txt.	Rule for Meta-ExternalFetcher
DuckAssistBot DuckDuckGo	User fetch. Fetches pages in real time for DuckDuckGo's AI-assisted answers. Blocking it does not affect DuckDuckGo search rankings.	Rule for DuckAssistBot
cohere-ai Cohere	Purpose unverified. Cohere's page says it does not crawl for generative AI training at this time. The cohere-ai token appears only in third-party lists.	Rule for cohere-ai
Diffbot Diffbot	AI search. Crawls for the Diffbot Knowledge Graph and web search services.	Rule for Diffbot
YouBot You.com	AI search. You.com's crawler for its AI search.	Rule for YouBot
Timpibot Timpi	Purpose unverified. Timpi's crawler. Timpi publishes no documentation we could find, so its purpose is unverified.	Rule for Timpibot
omgili Webz.io	Data collection. Webz.io's crawler, which feeds a web data product sold through its API. The operator page does not mention AI training.	Rule for omgili

Purposes come from each operator's own documentation, read October 29, 2026. Where an operator publishes nothing we could verify, the row says so.

Your current robots.txt (optional)

Sitemap address (optional)

Your robots.txt

# AI crawler rules

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Meta-ExternalFetcher
User-agent: DuckAssistBot
User-agent: Diffbot
User-agent: YouBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Bytespider
User-agent: CCBot
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: cohere-ai
User-agent: Timpibot
User-agent: omgili
Disallow: /

Built in your browser. Nothing you enter here is sent to us.

What does a robots.txt file do for AI crawlers?

A robots.txt file is a plain text file at the root of your site, such as yoursite.com/robots.txt. Crawlers read it before they fetch your pages. Each AI company publishes a crawler name, called a user agent, and says that crawler follows the rules you set for that name.

A rule is two lines: who it is for, and what they may do. For example, "User-agent: GPTBot" followed by "Disallow: /" asks OpenAI's training crawler to stay out of the whole site.

Robots.txt is a request, not a lock. The standard says so itself: "These rules are not a form of access authorization" (RFC 9309, read October 29, 2026). Most of the companies listed here say their crawlers honor it. User-triggered fetchers are the exception: OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores robots.txt, and Meta says Meta-ExternalFetcher may bypass it (OpenAI, Perplexity, Meta, read October 29, 2026). A crawler that ignores robots.txt can only be stopped at your server or CDN.

Which AI crawlers should I block?

Sort them by what they do.

Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) collect pages to train models, or set whether collected pages may be used for training. OpenAI says each setting is independent: a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot (OpenAI crawler docs, read October 29, 2026). Anthropic and Perplexity publish their own crawler names too (Anthropic, Perplexity).
AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the index that AI search quotes from. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers" (OpenAI crawler docs, read October 29, 2026). Block these and your pages are less likely to be quoted.
Data collection crawlers (CCBot, omgili) gather pages into data sets that other people can use. CCBot feeds the open Common Crawl archive.
User-fetch bots (ChatGPT-User, Claude-User, Perplexity-User) open one page because a person asked an AI tool to read it.

The first preset, allow AI search and block training, suits a site that wants to be cited but not used for training. It also blocks crawlers that collect data without a stated search purpose, and the two crawlers whose operators publish nothing we could verify (Bytespider and Timpibot). Our guide on how to block or allow ChatGPT in robots.txt covers the OpenAI names in detail. The dated snapshot of which well-known sites block AI crawlers shows what others chose.

What if my site already has a robots.txt?

Paste it into the existing rules box. The tool keeps your rules as written and adds the AI crawler rules after them. If you already name one of the crawlers, the tool tells you. Two groups for the same crawler are combined by the crawler, so read the result before you upload it.

How do I add the file to my site?

Pick allow AI search and block training, block all AI, or allow all. Then set any single crawler to allow, block or leave out.
Paste your current robots.txt and your sitemap address, if you have them. The tool keeps your rules as written.
Copy the text from the preview or download it as robots.txt.
Save it as robots.txt so it loads at yoursite.com/robots.txt. If your site builder has a robots.txt setting in its SEO tools, paste the text there.
Open the address in a browser, then run the free AI visibility checker to confirm what each crawler can read.

Then run the free AI visibility checker on your site.

Questions about the robots.txt generator

Does robots.txt stop AI companies from training on my content?

It asks the named crawlers to stay out. Most of the companies listed say their crawlers follow it, but user-triggered fetchers such as ChatGPT-User and Perplexity-User may not. It does not remove pages already collected, and it cannot stop a crawler that ignores the rules.

Will blocking GPTBot remove my site from ChatGPT?

Not on its own. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot. Per OpenAI's crawler docs (developers.openai.com/api/docs/bots), sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.

Should I name each crawler or use User-agent: *?

Name them. A wildcard rule would also block Googlebot and Bingbot, which most sites want to keep.

Does the tool store my robots.txt?

No. The file is built in your browser. Nothing you type here is sent to a server.

What if a crawler is missing from the list?

New crawlers appear and old ones are renamed. If you need one that is not listed, add its two lines to the existing rules box and the tool keeps them.

Next: give AI tools something to quote

Paperboy writes articles that answer the questions your buyers ask, with every fact linked to its source, FAQ and Article structured data, and publishes them to your site every week. Your first article is free.