What is Meta-WebIndexer?
Meta-WebIndexer is Meta's search crawler. Search crawlers build the index a search or answer product draws on.
| Operator | Meta |
|---|---|
| Purpose (operator wording) | search index |
| robots.txt token | Meta-WebIndexer |
| User agent string | meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers) (as printed on the doc page; the link is relative) ; also meta-webindexer/1.1 |
| Respects robots.txt | Not stated (robots.txt control implied) |
| IP ranges | Not stated by the operator |
| Last verified |
What else should I know about Meta-WebIndexer?
What is the Meta-WebIndexer user agent string?
This is the user agent string Meta documents for Meta-WebIndexer, as of October 29, 2026. Version numbers and details can change, so match on the token in robots.txt, not on the whole string.
meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers) (as printed on the doc page; the link is relative) ; also meta-webindexer/1.1How do I block Meta-WebIndexer in robots.txt?
Add this group to the robots.txt file at the root of your site. It asks Meta-WebIndexer to stay out of every page.
User-agent: meta-webindexer
Disallow: /To allow it explicitly, for example while a broader rule blocks other bots:
User-agent: meta-webindexer
Allow: /How do I opt out of Meta training but stay in its search?
Each Meta user agent has its own token, so you can set them one by one. This file blocks Meta-ExternalAgent and allows Meta-ExternalFetcher, Meta-WebIndexer. Every line comes from the rows in this dataset. Whether allowing a bot keeps you in a product is Meta's call, so check each crawler's page. Meta says Meta-ExternalFetcher may not follow robots.txt.
User-agent: meta-externalagent
Disallow: /
User-agent: meta-externalfetcher
Allow: /
User-agent: meta-webindexer
Allow: /Does blocking Meta-WebIndexer affect AI answers?
Meta: allowing it 'helps us cite and link to your content in Meta AI's responses.' Explicit statement that blocking removes citations: not stated.
Each other Meta user agent has its own robots.txt token, so a rule for Meta-WebIndexer does not cover them. See the related crawlers below.
Does Meta-WebIndexer follow robots.txt?
Not stated. Robots.txt control implied. Meta: 'Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses.'
Robots.txt is a request, not a lock (RFC 9309).
What IP addresses does Meta-WebIndexer use?
Not stated by the operator. Without a published list you cannot verify Meta-WebIndexer by address, so a rule at your server cannot rely on one.
Which other Meta crawlers are there?
What should I read next?
Next: give AI tools something to quote
Paperboy writes articles that answer the questions your buyers ask, with every fact linked to its source, FAQ and Article structured data, and publishes them to your site every week. Your first article is free.