AI crawler guide
ClaudeBot, Claude-User and Claude-SearchBot: What Each Does
Anthropic uses three user agents. ClaudeBot collects web content that may be used for model training, Claude-SearchBot improves the quality of search results in Claude, and Claude-User opens a page when a person asks Claude about it. Each can be allowed or blocked separately in robots.txt.
What Each Anthropic Bot Does
The three bots do different jobs, so blocking one does not block the others. If your goal is to be reachable when people use Claude, Claude-SearchBot and Claude-User matter most. Whether to allow ClaudeBot is a separate decision about training.
| User agent | Used for | Common choice |
|---|---|---|
| ClaudeBot | Collecting web content that may be used for model training | A policy choice: allow or block |
| Claude-SearchBot | Improving the quality of search results for Claude users | Allow, if you want to be findable in Claude search |
| Claude-User | Opening a page when a person asks Claude about it | Allow, if you want people to open your pages through Claude |
These descriptions follow Anthropic's published crawler documentation, which can change. Check Anthropic's current page before you rely on the details, including how its bots treat robots.txt.
How to Allow All Three Bots
To allow all three, add one group per bot to the robots.txt file at the root of your domain. If nothing in your file blocks them they are already allowed, and explicit rules just make your intent clear.
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /How to Block Training but Stay Reachable
To opt out of training while staying reachable, disallow ClaudeBot and allow the other two. Because each bot reads only its own group, one rule does not change the others.
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /Do not use a User-agent: * rule with Disallow: / to stop training. It blocks every crawler that has no group of its own, including the search bots you may want.
Why Claude Can Still Fail to Read Your Page
A page can be allowed in robots.txt and still be unreadable to Claude's bots, because a firewall or CDN rule can return a 403 or 429 error, and text that appears only after JavaScript runs looks empty to a crawler that reads raw HTML. Each cause needs a different fix.
- robots.txt disallows the bot, or a wildcard rule blocks everything.
- A firewall, CDN or bot-protection rule challenges or blocks the bot.
- The text you want quoted exists only after JavaScript runs.
The free GeoLint audit separates these three causes by testing robots.txt, the real server response and the raw HTML. See also robots.txt for AI bots.
ClaudeBot and Anthropic Crawlers: Common Questions
Should I block ClaudeBot?
That is a content-rights decision. As Anthropic describes it, ClaudeBot is for training data while Claude-SearchBot and Claude-User serve search and user requests, so you can block ClaudeBot and still allow the other two.
Does Anthropic say its bots follow robots.txt?
Anthropic's documentation says its crawlers follow robots.txt. Because policies change, read its current page, and use firewall rules too if you need a hard block.
How do I know if ClaudeBot is blocked on my website?
Check whether robots.txt disallows it and whether your server returns an error to it. GeoLint tests both layers for the main AI search bots and reports which one is the problem.
Check your own website for free
GeoLint asks each AI search bot for your page, loads it with and without JavaScript and shows what is blocking you. It is free, needs no signup and takes about 20 seconds.