What is CCBot?
CCBot is Common Crawl's crawler for another purpose. The operator describes a purpose that is neither AI training nor search. From the operator's wording: open web crawl repository; data is publicly available.
| Operator | Common Crawl |
|---|---|
| Purpose (operator wording) | other (open web crawl repository; data is publicly available) |
| robots.txt token | CCBot |
| User agent string | CCBot/2.0 (https://commoncrawl.org/faq/) |
| Respects robots.txt | Yes |
| IP ranges | https://index.commoncrawl.org/ccbot.json |
| Last verified |
What else should I know about CCBot?
What is the CCBot user agent string?
This is the user agent string Common Crawl documents for CCBot, as of October 29, 2026. Version numbers and details can change, so match on the token in robots.txt, not on the whole string.
CCBot/2.0 (https://commoncrawl.org/faq/)How do I block CCBot in robots.txt?
Add this group to the robots.txt file at the root of your site. It asks CCBot to stay out of every page.
User-agent: CCBot
Disallow: /To allow it explicitly, for example while a broader rule blocks other bots:
User-agent: CCBot
Allow: /Does blocking CCBot affect AI answers?
Common Crawl does not say what blocking CCBot changes. That is not stated.
Does CCBot follow robots.txt?
Yes. Common Crawl gives robots.txt opt-out instructions for CCBot. The page has no explicit sentence saying CCBot obeys robots.txt.
Robots.txt is a request, not a lock (RFC 9309).
What IP addresses does CCBot use?
Common Crawl publishes a list at https://index.commoncrawl.org/ccbot.json. We did not download or parse it. A request that claims to be CCBot but comes from outside that list is not CCBot.
What should I read next?
Next: give AI tools something to quote
Paperboy writes articles that answer the questions your buyers ask, with every fact linked to its source, FAQ and Article structured data, and publishes them to your site every week. Your first article is free.