Block ChatGPT-User in robots.txt? What works

You can write a robots.txt rule for ChatGPT-User, but it may not be honored. OpenAI's bot documentation says that because a user starts those fetches, "robots.txt rules may not apply." To stay in ChatGPT search while opting out of training, allow OAI-SearchBot and disallow GPTBot, which are the two tags OpenAI says webmasters use to manage how their content works with AI. This is per OpenAI documentation.

For the wider picture of which AI bots exist and what each one does, see our overview of blocking ChatGPT in robots.txt. This page is the how-to for the specific case of ChatGPT-User.

Quick version

• OAI-SearchBot: allow it if you want to be eligible to appear in ChatGPT search answers.
• GPTBot: disallow it if you want to signal that your content should not be used for training.
• ChatGPT-User: you can disallow it, but treat the rule as a request, not a guarantee.
• Hard blocking: if you must keep ChatGPT-User out, Elementera says the block has to happen in your firewall or CDN, not only in robots.txt.
• Timing: OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust.

The robots.txt file to use

This file allows OAI-SearchBot, blocks GPTBot, and adds a ChatGPT-User group so your preference is on record:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

The ChatGPT-User group uses the same form Elementera shows: a User-agent: ChatGPT-User line followed by Disallow: /. Put the file at the root of your domain, as /robots.txt.

If you want to keep ChatGPT-User unblocked in the file, delete the last group. Per Elementera, ChatGPT-User is allowed by default: if no group names it and no User-agent: * rule blocks it, it may read every page. That second condition matters. Check whether your file has a catch-all group before you assume what each bot will do.

What each OpenAI agent does

Three agents matter here, and TechWyse notes that their names matter because they do different jobs. A fourth, OAI-AdsBot, exists too.

OAI-AdsBot is a separate case. OpenAI says the data it collects is not used to train generative AI foundation models. This article does not cover it further.

How to block ChatGPT-User but allow OAI-SearchBot

This is the question behind most searches for this topic, and the file above answers it. OAI-SearchBot and ChatGPT-User are separate agents with separate groups. Search Engine Journal reports that, according to OpenAI's documentation, the agent that decides whether a site shows up in ChatGPT search results is OAI-SearchBot, not ChatGPT-User.

So a Disallow: / under ChatGPT-User does not, by itself, remove you from search results. And an Allow: / under OAI-SearchBot does not, by itself, stop ChatGPT-User fetches.

One warning from the other direction. If you disallow OAI-SearchBot, OpenAI says your site will not be shown in ChatGPT search answers. Do not copy a rule block from a guide without checking which agent name sits above each line.

Why the ChatGPT-User rule is only a request

OpenAI's wording is the key. In its documentation, OpenAI says: "Because these actions are initiated by a user, robots.txt rules may not apply." Note the word "may." It is not a promise that the rule is ignored, and it is not a promise that it is followed.

TechWyse explains the reasoning. A traditional crawler independently decides which URLs to visit at scale, and robots.txt was designed to communicate crawl preferences to systems operating in that mode. OpenAI explicitly says ChatGPT-User is not an automated web crawler.

Elementera describes the same thing in plain terms: it fetches a page when a person asks the assistant to look at it. SearchEngineWorld adds that when someone asks a question, ChatGPT may fetch a live web page to help answer and to provide a source link.

What the measurements show

There is third-party data on how this plays out. It comes from TollBit's State of the Bots report, as reported by Search Engine Journal:

• ChatGPT-User, Bytespider and Youbot each accessed disallowed pages on nearly half of the European sites that had explicitly listed them.
• About 15% of identified AI page-fetchers reached URLs that those European sites had marked as disallowed.

These figures cover European sites in one report. They do not tell you what will happen on your site. They do show that a disallow line for this agent should not be treated as a lock.

Anthropic takes a different position: it states that all three of its bots respect robots.txt. Perplexity says Perplexity-User generally ignores the file for the same user-initiated reason OpenAI gives. Rules differ by company, so do not assume one bot's behavior carries over to another.

How common is a ChatGPT-User block?

Not very, at least among the sites Elementera measured. In its sample of 569 startup websites, 93 sites (16%) name ChatGPT-User in their robots.txt files. Only 3 of 569 sites (0.5%) block it on at least one path, and 1 of 569 (0.2%) blocks it on all six common paths it checked. That is a sample of startup sites, not the web as a whole.

Our view: this is a reminder that many site owners are making this decision by default rather than on purpose. This is our judgment, not a sourced fact.

If you need a hard block

robots.txt cannot enforce anything. TechWyse puts it this way: it has always been a protocol for communicating crawler rules, not a network security boundary. If ChatGPT-User access is a real problem for your site, the stronger controls sit elsewhere.

1. Check your logs. Playwire writes that the robots.txt file shows what you asked for, while server logs and CDN records show what happened, and the two are not always the same. Playwire sells ad revenue optimization software for publishers, so read its framing with that in mind. The log point itself is simple and easy to check on your own site.
2. Verify the source of requests. User agents can be faked. Elementera advises matching the request's IP address against OpenAI's published list at openai.com/chatgpt-user.json. TechWyse notes that OpenAI publishes IP ranges for ChatGPT-User, OAI-SearchBot and GPTBot, which gives infrastructure teams another signal to inspect.
3. Block at the firewall or CDN. Per Elementera, if you must keep ChatGPT-User out, the block has to happen in your firewall or CDN, not only in robots.txt.
4. Weigh the trade. SearchEngineWorld says blocking ChatGPT-User prevents ChatGPT from fetching your pages during user requests. That also means a person asking ChatGPT about your page may get no live content from it. Whether that is acceptable depends on your site.

Check that your changes worked

1. Save the file at yourdomain.com/robots.txt and open it in a browser to confirm it loads.
2. Wait. OpenAI says it can take about 24 hours for its systems to adjust after a robots.txt update.
3. Review your server logs for the three agent names over the following days.
4. Compare the log lines against the IP list from step 2 above before you trust any user-agent string.

User-agent strings also change. OpenAI's example for GPTBot shows version 1.4 and notes that the version number may change. Match on the agent name, not the full string.

If you want a quick look at how your site appears to AI tools, Paperboy has a free AI visibility checker. It does not change your robots.txt, and it makes no promise about citations or traffic.

What we would do

Our view, not a sourced fact: for most small business sites, the file above is a sensible default. It keeps the door open to ChatGPT search, signals a training opt-out, and records a ChatGPT-User preference. Then treat the ChatGPT-User line as a preference and do not build policy on it. If a site holds content that truly must not be fetched, use access controls, not a text file.

For other questions on AI bots and search, browse the Paperboy blog.