Who blocks AI crawlers? A dated robots.txt snapshot

Updated October 26, 2026

On October 26, 2026, 40.0% of the 55 well-known sites we could fetch block GPTBot in robots.txt, 49.1% block ClaudeBot and 36.4% block PerplexityBot. Block at least one of the three: 49.1%. About half the list is news and media, so this is not a small-business sample.

Counts come straight from the CSV behind the table below. "Blocked" means the file has a group for that bot with Disallow: /. 60 sites were on the list and 55 returned a robots.txt we could read.

Summary counts

Out of 55 measured sites:

• 5 sites block ClaudeBot but not GPTBot: washingtonpost.com, theguardian.com, businessinsider.com, investopedia.com, allrecipes.com.
• The 5 sites whose * group disallows / are reuters.com, wsj.com, quora.com, reddit.com, yelp.com. They are not counted as "blocked" in the bot columns. reuters.com and reddit.com name none of the three bots, so the wildcard is the only thing closing them to these crawlers.
• We fetched the same 60 sites on October 25, 2026 as a baseline. No site changed its GPTBot, ClaudeBot or PerplexityBot result between the two fetches.

Of the 29 measured sites that are news, media or other content publishers, 17 block GPTBot, 22 block ClaudeBot and 17 block PerplexityBot. Of the other 26 (retail, marketplaces, home services, finance, platforms and SEO tools), 5, 5 and 3 do. The list leans toward publishers, which is why the headline numbers should not be read as typical.

Full table: all 55 measured sites

Fetched October 26, 2026. no_specific_rule means no group names that bot. The last column says whether the * group disallows /.

Not measured (5): bestbuy.com (timeout), godaddy.com (403), mayoclinic.org (403), foodnetwork.com (403), stackoverflow.com (418). Each failed on the first fetch and again with a second browser user agent, so they are left out of every count and percentage.

Download the CSV (columns: date, site, http_status, robots_bytes, GPTBot, ClaudeBot, PerplexityBot, wildcard_disallow_root, note).

What we measured and how

We took 60 well-known US-facing sites (news, retail, marketplaces, home services, finance, health, SEO and web platforms) and requested https://SITE/robots.txt once each on October 26, 2026. The request used the user agent Mozilla/5.0 (compatible; PaperboyResearch/1.0). If it failed, we retried once with a desktop Chrome user agent. Then we read the user-agent groups:

• blocked: a group names the bot and contains Disallow: / with no Allow: /.
• partial: a group names the bot and disallows some other paths.
• allowed: a group names the bot and disallows nothing.
• no_specific_rule: no group names the bot.

Reproduce one site with curl:

curl -sL -m 20 -A "Mozilla/5.0 (compatible; PaperboyResearch/1.0)" https://nytimes.com/robots.txt \
  | grep -i -A3 -E "^user-agent: *(GPTBot|ClaudeBot|PerplexityBot)"

Loop over the sites in the table and read the output by eye, or use the script we used. For what each bot is and how to set the rules yourself, see our guide on how to block ChatGPT with robots.txt.

Caveats

• Not a small-business sample. The list is 60 well-known US-facing sites. It says nothing about small businesses and should not be read as the top 100 anything. Roughly half are news, media and other content publishers (31 of 60 by our count), and those block AI crawlers more than typical sites.
• robots.txt is a request, not enforcement. Compliant crawlers follow it. Some sites also block at the CDN or firewall, which this method cannot see. See RFC 9309, which says the rules are not a form of access authorization.
• Parser limits. Our parser does not apply * inheritance, wildcards in paths, or Allow-over-Disallow specificity beyond Allow: /. So partial and no_specific_rule do not guarantee a bot can or cannot reach a page.
• One day, one fetch, one network. Each site was fetched once from one network on October 26, 2026. The 5 sites that returned 403, 418 or a timeout may serve different content to other clients. Robots.txt files change, so treat this as a dated snapshot.
• Three bots only. We did not test OAI-SearchBot, Claude-User, Perplexity-User, Google-Extended or other crawlers. What those crawlers do, and the lines that block them, are on the AI crawlers list.

Check your own site

Run the free AI visibility checker to see how your site looks to AI search. Paperboy writes, fact-checks and publishes SEO articles, and your first article is free.