AI crawler guide
PerplexityBot and Perplexity-User: What Each Does
Perplexity uses two user agents. PerplexityBot builds the search index that Perplexity draws on, and Perplexity-User visits a page when a person asks a question that needs it. They behave differently, so allow each one deliberately and test the real response from your server.
What Each Perplexity Bot Does
PerplexityBot is the indexing bot, so blocking it can keep your pages out of Perplexity's search results. Perplexity-User is a user-triggered fetcher that opens a page during a live question.
| User agent | Used for | Common choice |
|---|---|---|
| PerplexityBot | Building the index that Perplexity search draws on | Allow, if you want to appear in Perplexity results |
| Perplexity-User | Visiting a page when a person's question needs it | Allow in your firewall, so people can open your pages |
These descriptions follow Perplexity's published bot documentation, which can change. That page also explains how each bot treats robots.txt and whether it publishes IP address lists. Read it before you write firewall rules.
How to Allow Both Bots
To allow both, add a group for each bot to robots.txt. Allowing them in robots.txt is only the first layer, because a firewall can still block them.
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /Why a Firewall Can Block Perplexity Even When robots.txt Allows It
robots.txt is a request, while your firewall or CDN decides what a bot actually receives. Bot-protection rules, rate limits and challenge pages can return a 403 or 429 to AI crawlers even when robots.txt says yes.
Read robots.txt
Confirm that neither bot is disallowed and that no User-agent: * group blocks the whole site.
Check your CDN and firewall
Look for bot-protection or managed rules that challenge or block these user agents, and allow the ones you want.
Test as the bot
Run the free GeoLint audit to see what robots.txt says and what the server returns for each AI search bot.
For the access basics across all AI crawlers, read robots.txt for AI bots.
PerplexityBot: Common Questions
Does blocking PerplexityBot remove me from Perplexity?
It can. PerplexityBot builds the index that Perplexity search draws on, so blocking it can keep your pages from appearing in results. Check Perplexity's current documentation for exactly how it is used.
What is Perplexity-User?
Perplexity describes Perplexity-User as the agent that visits a page when a person's question needs it. Because a person triggers it, Perplexity's documentation explains how it treats robots.txt, so read that page for the current rule.
How do I test if Perplexity can read my website?
Test two layers: what robots.txt says and what your server actually returns to the bot. GeoLint checks both and also tests whether your text is in the HTML without JavaScript.
Check your own website for free
GeoLint asks each AI search bot for your page, loads it with and without JavaScript and shows what is blocking you. It is free, needs no signup and takes about 20 seconds.