Block ClaudeBot in robots.txt if AI training is your concern, and leave Claude-User and Claude-SearchBot alone. Per Anthropic's documentation, ClaudeBot is the bot that collects web content that could contribute to model training. The other two serve user questions and search quality, and disabling either one stops Anthropic's system from retrieving or indexing your content for those uses.
Quick version
- Block training only: disallow
ClaudeBot, allowClaude-UserandClaude-SearchBot. - Block everything from Anthropic: disallow all three user agents, one group each.
- Allow everything: add no rule for these bots, or add explicit Allow lines.
- Do not rely on IP blocking. Anthropic says it may not work correctly or persistently guarantee an opt-out.
Running the same decision for OpenAI's bots? See our guide to blocking ChatGPT in robots.txt.
The three Anthropic bots
Anthropic's help page says it uses three robots and lets site owners enable the ones they want and limit the ones they do not. Each carries a separate user-agent string, so each can get its own rule.
The descriptions are Anthropic's own words. The effect column comes from Anthropic's documentation as quoted by Search Engine Roundtable, and from Anthropic's help page for ClaudeBot.
Which bots should you block?
The answer depends on what you want from Claude. There are three common positions.
You object to training, but want to stay readable
Block ClaudeBot only. Search Engine Journal puts the limit plainly: blocking ClaudeBot stops training data collection but does nothing about Claude-SearchBot or Claude-User. That is the point. The other two bots keep working, so Claude can still fetch your pages when someone asks about them.
Our view: for most small business sites that publish to be found, this is the sensible default. It is our judgment, not a sourced fact. The reasoning is that a visitor who asks Claude about your business needs Claude to be able to read your site.
You want no Anthropic access at all
Block all three. That means Claude-User requests for your page fail, and Claude-SearchBot cannot index it. Menra, the source for that point, sells citation tracking and AI engine monitoring, so treat its framing with that in mind. The mechanics match what Anthropic's own documentation says about disabling each bot.
You are happy to be used
Add no rules. Anthropic's bots follow the rules in your file, and with no rule for them there is nothing to follow.
What blocking might cost you
There is no clean number for ClaudeBot alone. One study reported by PPC Land, from Rutgers Business School and The Wharton School, found that publishers who blocked AI crawlers via robots.txt saw total traffic fall by 23%. That figure covers AI crawlers as a group, not ClaudeBot specifically, and it concerns publishers, so it may not apply to your site. We cite it as a reason to block with intent rather than by reflex, not as a forecast.
For scale, a BuzzStream study covered by Search Engine Journal found that 79% of top news sites block at least one AI training bot. The same coverage says 71% also block at least one retrieval or search bot. Many large publishers are blocking both kinds, but your goals may differ from theirs.
The robots.txt lines
The file lives at the root of your domain, at /robots.txt. Each bot gets its own group. robots.txt became a formal standard, RFC9309, in 2022.
Block training, allow the other two
This is the pattern Menra shows for allowing Claude-SearchBot and Claude-User while disallowing ClaudeBot:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
The first group is the same block Anthropic gives for ClaudeBot: User-agent: ClaudeBot, then Disallow: /.
Block all three
This is the same syntax applied to each user agent. It is our adaptation, not a block copied from Anthropic:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
Block a section, not the whole site
Replace the slash with a path. Here the path is a placeholder, not a real recommendation:
User-agent: ClaudeBot
Disallow: /members/
Slow a bot down instead
Anthropic says that to limit crawling activity, it supports the non-standard Crawl-delay extension to robots.txt. Add a Crawl-delay line to the relevant group with a value that suits your server. Because the extension is non-standard, treat it as a request, not a guarantee.
What robots.txt does not do
- It does not undo past collection. cside, which sells browser-layer AI agent detection software, says blocking ClaudeBot prevents new content from entering future training runs but does not remove previously collected content from existing Claude models. Anthropic's own wording is forward-looking too: restricting ClaudeBot signals that future materials should be excluded.
- It does not cover Claude-powered agents browsing in a real browser. cside also says blocking ClaudeBot does not prevent Claude-powered agents from visiting your site.
- It is not enforcement. robots.txt is a request. Anthropic says its bots honor industry standard directives in robots.txt, and Search Engine Journal reports that this includes Claude-User. Still, PPC Land notes that Reddit's lawsuit, filed on June 4, 2025, alleged that Anthropic continued accessing its platform more than 100,000 times after publicly claiming it had stopped. That is an allegation in a lawsuit, not a finding. If you need hard enforcement, you need server-level controls.
- Do not block by IP address alone. Anthropic warns that this may not work correctly or persistently guarantee an opt-out. PPC Land adds that it impedes Anthropic's ability to read your robots.txt file.
- User-agent strings can be faked. Menra says scrapers routinely impersonate AI crawlers, and that verification should be IP-based. Anthropic publishes ClaudeBot IP ranges, according to cside.
Old rules you may still have
If your robots.txt predates the three-bot setup, check it. Search Engine Journal reports that before ClaudeBot, Anthropic operated under the Claude-Web and Anthropic-AI user agents, both now deprecated. Rules for those names do not control the three current bots, so add rules for the names above.
Check your file
- Open
yourdomain.com/robots.txtin a browser. - Confirm each of the three user agents is spelled exactly:
ClaudeBot,Claude-User,Claude-SearchBot. - Confirm each group does what you intend: Disallow to block, Allow or no rule to permit.
- Look for older Claude-Web or Anthropic-AI lines and decide whether to keep them.
- Re-read Anthropic's help page now and then, since the bot list has changed before.
Paperboy also has a free AI check if you want a quick look at how your site presents to AI tools. Nothing in robots.txt promises rankings, citations or traffic either way. It only controls which Anthropic bots may read your pages. For more guides like this one, see the blog.
Frequently asked questions
Does ClaudeBot obey robots.txt?
Anthropic says its bots honor industry standard directives in robots.txt. Search Engine Journal reports that this covers all three bots, including Claude-User.
If you block ClaudeBot, will Claude stop mentioning your site?
Not necessarily. Blocking ClaudeBot does nothing about Claude-SearchBot or Claude-User, and those are the bots tied to search and to user questions. We cannot promise any outcome in Claude's answers either way.
Can you slow ClaudeBot down instead of blocking it?
Anthropic says it supports the non-standard Crawl-delay extension to robots.txt. Add the line to the bot's group in your file.
Is blocking Anthropic's IP addresses a good alternative?
Anthropic says that approach may not work correctly or persistently guarantee an opt-out. Use robots.txt rules as the main method.
Do Claude-Web and Anthropic-AI still matter?
Search Engine Journal says both user agents are now deprecated. Add rules for ClaudeBot, Claude-User and Claude-SearchBot instead.