AI crawler guide
GPTBot, OAI-SearchBot and ChatGPT-User: What Each Does
OpenAI uses three user agents. GPTBot gathers content that may be used to train models, OAI-SearchBot finds pages for ChatGPT search results, and ChatGPT-User fetches a page when a person asks ChatGPT to open it. You can set a rule for each one separately.
What Each OpenAI Bot Does
The three bots have different jobs, so allowing or blocking them has different effects. If you want to appear in ChatGPT search, OAI-SearchBot is the one that matters most. Whether to allow GPTBot is a separate choice about model training.
| User agent | Used for | Common choice |
|---|---|---|
| GPTBot | Collecting content that may be used to train OpenAI models | A policy choice: allow or block |
| OAI-SearchBot | Finding and listing pages in ChatGPT search results | Allow, if you want to appear in ChatGPT search |
| ChatGPT-User | Fetching a page when a person asks ChatGPT to open it | Allow, if you want people to open your pages through ChatGPT |
These descriptions follow OpenAI's published crawler documentation, which can change. Check OpenAI's current bot page before you rely on the details.
How to Allow All Three Bots
To allow all three, add one group per bot to the robots.txt file at the root of your domain. If your robots.txt has no rule blocking these bots, they are already allowed by default, and an explicit Allow just makes your intent clear.
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /Save the file as robots.txt in the root of your site, for example https://example.com/robots.txt. Rules only apply to the host that serves the file.
How to Block Training but Stay in ChatGPT Search
To opt out of model training while staying eligible for ChatGPT search, disallow GPTBot and allow OAI-SearchBot. Because the bots are separate, one rule does not affect the other.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Blocking GPTBot is a content-rights decision, not a visibility fix. If your goal is to appear in ChatGPT search, check OAI-SearchBot first.
Why a Bot Can Still Be Blocked When robots.txt Allows It
A bot can be allowed in robots.txt and still be blocked, because robots.txt is a request and your firewall or CDN decides what actually happens. Bot-protection rules, rate limits and challenge pages can return a 403 or 429 error to AI crawlers even when robots.txt says yes.
Read your robots.txt
Open /robots.txt on your domain and confirm that no User-agent: * group disallows the whole site and that none of the three bot names is disallowed.
Check your CDN and firewall
Look at bot-protection, managed rules and any setting that blocks AI crawlers in your CDN or hosting dashboard. Make sure they do not challenge or block the bots you want. If you use Cloudflare, follow the Cloudflare checklist.
Test the real response
The only reliable proof is a request made as the bot. GeoLint tests both layers for each bot and reports what robots.txt says and what the server returned.
GPTBot and ChatGPT Crawlers: Common Questions
Should I block GPTBot?
That depends on whether you are comfortable with your content being used to train models. According to OpenAI's description, GPTBot is separate from OAI-SearchBot, which is used for ChatGPT search, so blocking GPTBot is a different decision from staying visible in search.
Does ChatGPT-User follow robots.txt?
OpenAI describes ChatGPT-User as a fetcher that acts when a person asks ChatGPT to open a page. How it treats robots.txt is set by OpenAI and can change, so check its current documentation. Allowing it in both robots.txt and your firewall is the safe way to let people open your pages.
How do I know if GPTBot is blocked on my website?
Check two things: whether robots.txt disallows it, and whether your server or firewall returns an error to it. GeoLint tests both for each OpenAI bot and tells you which layer is the problem.
Why can ChatGPT not read my website?
The usual causes are a robots.txt rule that blocks the bot, a firewall or CDN rule that returns 403 or 429, or text that only appears after JavaScript runs. Each has a different fix, so test before changing anything.
Check your own website for free
GeoLint asks each AI search bot for your page, loads it with and without JavaScript and shows what is blocking you. It is free, needs no signup and takes about 20 seconds.