AI crawler guide
robots.txt for AI Bots: GPTBot, ClaudeBot, PerplexityBot and More
To control AI crawlers, add a User-agent group for each bot to the robots.txt file at the root of your domain, then use Allow or Disallow lines. Each AI company publishes its own bot names, so you list GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended separately.
The AI Bot Names to Know
Most AI companies run more than one crawler, and each has a different purpose. Training crawlers collect content for models, search crawlers build an index for AI search, and user-triggered fetchers open a page when a person asks for it.
| User agent | Company | Used for |
|---|---|---|
| GPTBot | OpenAI | Collecting content that may be used for model training |
| OAI-SearchBot | OpenAI | Listing pages in ChatGPT search |
| ChatGPT-User | OpenAI | Opening a page when a person asks ChatGPT to |
| ClaudeBot | Anthropic | Collecting content that may be used for model training |
| Claude-SearchBot | Anthropic | Improving search result quality in Claude |
| Claude-User | Anthropic | Opening a page when a person asks Claude to |
| PerplexityBot | Perplexity | Building the Perplexity search index |
| Perplexity-User | Perplexity | Opening a page when a person asks Perplexity to |
| Google-Extended | A robots.txt token that controls whether content Google crawls can be used for its Gemini AI models |
Names and purposes follow each company's published crawler documentation and can change. Google-Extended is not a separate crawler and does not change how Googlebot indexes your site for normal search. Check each vendor's current page before relying on this table.
Example: Allow AI Search, Block Model Training
To stay visible in AI search while opting out of training, allow the search bots and disallow the training bots and the Google-Extended token. Keep each bot in its own group so the rules do not collide.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /This is one policy, not the only right one. Whether to allow training crawlers is a business decision about how your content may be used.
Example: Allow Every Bot
To allow every crawler, use a single wildcard group with an empty Disallow line, or leave robots.txt without any blocking rules. Both mean the same thing to a well-behaved bot.
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlCommon robots.txt Mistakes That Block AI Bots
Most accidental blocks come from five mistakes, and the first is the most damaging: a leftover rule that disallows everything.
- A User-agent: * group with Disallow: / blocks every crawler that has no group of its own.
- A misspelled bot name, so the rule never matches the bot you meant.
- The file in the wrong place. It must be at /robots.txt on the exact host, and a subdomain needs its own file.
- A staging robots.txt that blocks everything being copied to the live site.
- Assuming robots.txt enforces the rule. It is a request that well-behaved bots follow.
robots.txt Is Not Access Control
robots.txt tells well-behaved crawlers what you want, but it cannot stop a bot or let one in. Your firewall, CDN and server decide what actually happens, so a bot allowed in robots.txt can still receive a 403 or a challenge page.
That gap is a common reason a site is invisible to AI search even though its robots.txt looks fine. See why a bot can still be blocked, the Cloudflare checklist and test the real response with the free GeoLint audit.
Related GEO Guides
- GPTBot, OAI-SearchBot and ChatGPT-User: What Each Does
- ClaudeBot, Claude-User and Claude-SearchBot: What Each Does
- PerplexityBot and Perplexity-User: What Each Does
- What Is llms.txt? A Plain-English Guide With an Example
- Is Your Website Blocking AI Crawlers? How to Check
- We Fix Your Website So AI Search Tools Can Read It
- All GEO guides
robots.txt and AI Crawlers: Common Questions
Where does the robots.txt file go?
It goes in the root of the host it controls, for example https://example.com/robots.txt. Rules apply only to that host, so a subdomain such as blog.example.com needs its own robots.txt file.
Does blocking GPTBot hide my site from ChatGPT?
Not by itself, according to OpenAI's description of its bots. GPTBot is separate from OAI-SearchBot, which is used for ChatGPT search. To check whether ChatGPT search can reach you, look at OAI-SearchBot and at your firewall.
Do all AI bots obey robots.txt?
Not necessarily. robots.txt is voluntary, and reputable companies say their crawlers follow it, but it cannot force a bot to comply. If you need to keep a bot out, use your firewall or server rules as well.
How do I test my robots.txt for AI bots?
Open yourdomain.com/robots.txt to read the rules, then test what each bot actually receives. GeoLint does both for the main AI search bots and shows any difference between the two.
Check your own website for free
GeoLint asks each AI search bot for your page, loads it with and without JavaScript and shows what is blocking you. It is free, needs no signup and takes about 20 seconds.