Small business owners: allow PerplexityBot in robots.txt if you want your pages to be able to appear in Perplexity search. Per Perplexity documentation, checked October 24, 2026, PerplexityBot is designed to surface and link websites in search results on Perplexity, and it is not used to crawl content for AI foundation models. Perplexity-User is a separate agent, and the same documentation says it generally ignores robots.txt rules.
Perplexity also recommends allowing PerplexityBot. We know this from Cloute, an AI visibility company that sells services to local businesses, which quotes Perplexity as recommending it so a site can appear in its search results. Check Perplexity's own page before you rely on that wording.
Quick version
- Want to appear in Perplexity search: allow PerplexityBot. Perplexity says it is built to surface and link sites.
- Do not want to be crawled by it: put
Disallow: /underUser-agent: PerplexityBot, as this robots.txt guide describes. - Perplexity-User is separate. Perplexity says it generally ignores robots.txt rules because a user requested the fetch.
- Robots.txt is a request, not a lock. Verify real traffic against Perplexity's published IP lists.
- Changes are not instant. Perplexity notes settings may take up to 24 hours to be reflected.
If you are making the same decision for OpenAI's crawlers, our guide to blocking ChatGPT in robots.txt covers that separately.
What PerplexityBot does
Perplexity lists two agents. PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models. That is Perplexity's statement about its own bot. It is a claim you can read, not something we have tested.
The full user-agent string, per Perplexity's documentation, is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot). In robots.txt you only need the token PerplexityBot.
One technical point worth knowing: PerplexityBot does not render JavaScript, according to Hybridranking's explainer. If your key content only appears after scripts run, a crawler that skips JavaScript may not see it. Menra, which sells citation tracking tooling, makes a related point: a JavaScript challenge served to PerplexityBot is functionally a block because the crawler will not execute it.
The robots.txt lines to allow PerplexityBot
The simplest allow rule is an explicit group:
User-agent: PerplexityBot
Allow: /
If your site already allows compliant crawlers by default, an explicit group is optional. Adding one can make your policy easier to audit and maintain.
There is one trap. The specific PerplexityBot group is evaluated on its own and does not automatically inherit wildcard restrictions. If you have a broad rule like User-agent: * with several Disallow lines, and you add a PerplexityBot group, repeat any paths you still want hidden inside that group. Here is a made-up example, not real data:
User-agent: *
Disallow: /private/
User-agent: PerplexityBot
Allow: /
Disallow: /private/
Without the second Disallow, PerplexityBot would follow only its own group.
The robots.txt lines to block PerplexityBot
To exclude the whole host, use Disallow: / under User-agent: PerplexityBot:
User-agent: PerplexityBot
Disallow: /
To block only part of the site, list the paths instead of the single slash.
Know what blocking costs. Menra says blocking PerplexityBot removes your pages from Perplexity's search index, which eliminates most citation opportunities. Menra sells citation tracking, so treat its framing with some care. It matches what Perplexity says the bot is for: surfacing and linking sites in search results.
How Perplexity-User is different
Perplexity-User is not a crawler in the search sense. A robots.txt guide puts it this way: PerplexityBot and Perplexity-User have separate roles and should be evaluated independently.
| PerplexityBot | Perplexity-User | |
|---|---|---|
| Purpose | Surface and link websites in Perplexity search results | Visits a web page when a user asks Perplexity a question |
| Used for AI model training? | No, per Perplexity | No, per Perplexity |
| Robots.txt | Follows it, per Roborank | Generally ignores it, since a user requested the fetch |
| IP list | perplexitybot.json | perplexity-user.json |
The practical result: blocking PerplexityBot does not stop every Perplexity visit. Menra notes that Perplexity-User can still fetch a URL when a user explicitly asks about it or pastes a link. If you need to stop that, robots.txt is not the tool. You would need server or firewall rules, and Perplexity-User's published IP addresses are the place to start.
What robots.txt cannot guarantee
Robots.txt depends on the crawler choosing to obey it. Two caveats are worth reading before you decide.
- Compliance is disputed. Hybridranking's explainer cites TollBit's State of the Bots report for the first half of 2026, saying PerplexityBot ignored explicit robots.txt instructions on 42% of its scrapes. We have not seen the TollBit report itself, so treat that as a secondhand figure.
- Undeclared crawlers were reported in 2025. Roborank says Cloudflare reported in August 2025 that Perplexity used undeclared, Chrome-spoofing crawlers. Hybridranking adds that Cloudflare removed Perplexity from its list of verified bots after that report. Perplexity's own documentation, linked above, describes only the two declared agents.
User-agent names can also be faked by anyone. One guide advises that for firewall rules or log analysis you should compare the request source with Perplexity's current published IP lists rather than trusting the header alone. Perplexity publishes PerplexityBot IP addresses and Perplexity-User IP addresses as JSON files. Hybridranking reports that on October 12, 2026, perplexitybot.json listed 8 IPv4 prefixes and perplexity-user.json listed 4. Those lists change, so read the current files rather than copying numbers.
If you want a hard block, a firewall rule on verified IP ranges is stronger than a robots.txt line. If you only want to signal your preference to a crawler that complies, robots.txt is enough.
Should a small business allow it?
This section is opinion.
Our view: most small businesses that want to be found through AI answer tools should allow PerplexityBot. The bot's stated job is to surface and link websites in search results, and Perplexity says it is not used to crawl content for AI foundation models. So the usual worry about blocking a training crawler does not apply to this agent, if Perplexity's description is accurate. This is our judgment, not a sourced fact.
There are fair reasons to block. You may run a members-only site, a site with licensed content, or a site where you simply do not want third-party use. Those are your call to make.
Allowing the bot is not a promise of anything. Hybridranking's article reports that Perplexity crawled 194.8 pages for every visitor it referred in July 2025. That is one month of data, from July 2025, and it shows crawling and referral traffic are not the same thing.
There is some evidence that being read matters for mentions. Cloute, which sells AI visibility services, reports that Perplexity named a business 81.3% of the time when it had read one of the business's own pages and 16.7% when it had not. That is one vendor's measurement from its own data. It does not show that allowing a bot causes a mention, and it does not guarantee a citation for your site.
A short checklist
- Open
yoursite.com/robots.txtand look for anyUser-agent: *rules and any existing PerplexityBot group. - Decide: allow, block part of the site, or block all of it.
- Add the group. If you also keep wildcard
Disallowlines, repeat the paths you want hidden inside the PerplexityBot group. - Check that your content is readable without JavaScript, since PerplexityBot does not render it.
- Keep an accurate sitemap. Menra says Perplexity favors recently updated pages and respects lastmod values in XML sitemaps.
- Wait. Changes may take up to 24 hours to reach Perplexity's systems.
- Check server logs against the published IP lists, not the user-agent name alone.
If you want to see how your site looks to AI tools before you change anything, Paperboy has a free AI visibility check. For what tends to help pages get cited once crawlers can reach them, see our post on getting your site cited by ChatGPT and Perplexity.
Sources and dates
The core claims about what PerplexityBot and Perplexity-User do come from Perplexity's crawler documentation, checked October 24, 2026. Robots.txt syntax comes from a robots.txt guide. Compliance figures come from a secondary article that cites TollBit and Cloudflare. Two sources we cite sell related services: Menra (citation tracking) and Cloute (AI visibility). We have said so where we used them.
Frequently asked questions
Does PerplexityBot train AI models on my content?
Perplexity's documentation says PerplexityBot is not used to crawl content for AI foundation models. That is Perplexity's own statement, so it rests on trusting the company's description.
Will Perplexity-User obey my robots.txt?
Generally no. Perplexity says since a user requested the fetch, this fetcher generally ignores robots.txt rules. To control it you need server or firewall measures, not a robots.txt line.
How long until a robots.txt change takes effect for Perplexity?
One guide says Perplexity notes settings work independently and may take up to 24 hours to be reflected. Allow a day before you test the result.
Do I need an explicit Allow rule for PerplexityBot?
Not always. If your site already allows compliant crawlers by default, an explicit group is optional, though it can make your policy easier to audit.
How can I tell if a visit is really from PerplexityBot?
Do not trust the user-agent name alone, because it can be spoofed. Compare the request's IP with the published PerplexityBot IP list.