Where do I download the AI crawler dataset?
| JSON (canonical) | ai-crawlers.json |
|---|---|
| CSV | ai-crawlers.csv. Same rows. Tokens and sources are joined with " | ". |
| Rows | 34 user agents, 16 operators |
| Fields | slug, name, operator, tokens, ua_string, purpose, respects_robots, respects_robots_source, blocking_effect, robots_block, robots_allow, ip_ranges_url, docs_url, sources, last_verified, notes |
| License | CC BY 4.0: reuse it freely and credit Paperboy. |
| Last updated | |
| Browse it | AI crawlers list, with one page per user agent |
How do I cite it?
Cite as: Paperboy AI Crawler Dataset, paperboyseo.com/ai-crawlers, updated October 29, 2026
Every row comes from the operator's own documentation, read October 29, 2026. Where an operator says nothing, the field says "not stated" or "unverified". We did not download or parse the IP range files. A wrong or missing row? Tell us on the contact page and cite the operator page that shows it.
What changed in the dataset?
Is there data on who blocks AI crawlers?
Yes. On October 26, 2026 we fetched robots.txt from 60 well-known sites and could read 55 of them. The CSV says whether each blocks GPTBot, ClaudeBot and PerplexityBot. It is one fetch from one network, so treat it as a dated snapshot.
- robots-ai-crawlers-2026-10-26.csv (CC BY 4.0, cite as Paperboy)
- Who blocks AI crawlers: the robots.txt snapshot, with the method and the limits
Where do I go from here?
Next: give AI tools something to quote
Paperboy writes articles that answer the questions your buyers ask, with every fact linked to its source, FAQ and Article structured data, and publishes them to your site every week. Your first article is free.
How do I cite this page?
<a href="https://paperboyseo.com/data">Paperboy open data: the AI crawler dataset</a> (Paperboy)License: CC BY 4.0. Use it, share it and adapt it, and credit Paperboy with a link to this page.