What is Diffbot-User?
Diffbot-User is Diffbot's user-triggered fetcher. A user-triggered fetcher runs when a person asks an AI product to open a page, not on a schedule.
| Operator | Diffbot |
|---|---|
| Purpose (operator wording) | user-triggered fetch |
| robots.txt token | Diffbot-User |
| User agent string | Not stated by the operator |
| Respects robots.txt | Not stated |
| IP ranges | Not stated by the operator |
| Last verified |
What else should I know about Diffbot-User?
What is the Diffbot-User user agent string?
Diffbot does not publish a full user agent string for Diffbot-User. The robots.txt token to use is Diffbot-User.
How do I block Diffbot-User in robots.txt?
Add this group to the robots.txt file at the root of your site. It asks Diffbot-User to stay out of every page.
User-agent: Diffbot-User
Disallow: /To allow it explicitly, for example while a broader rule blocks other bots:
User-agent: Diffbot-User
Allow: /Does blocking Diffbot-User affect AI answers?
Diffbot does not say what blocking Diffbot-User changes. That is not stated.
Each other Diffbot user agent has its own robots.txt token, so a rule for Diffbot-User does not cover them. See the related crawlers below.
Does Diffbot-User follow robots.txt?
Not stated. Diffbot docs describe Diffbot-User as requests 'on behalf of a human user browsing a URL using Diffbot software' and do not state its robots.txt behavior.
Robots.txt is a request, not a lock (RFC 9309).
What IP addresses does Diffbot-User use?
Not stated by the operator. Without a published list you cannot verify Diffbot-User by address, so a rule at your server cannot rely on one.
Which other Diffbot crawlers are there?
What should I read next?
Next: give AI tools something to quote
Paperboy writes articles that answer the questions your buyers ask, with every fact linked to its source, FAQ and Article structured data, and publishes them to your site every week. Your first article is free.