How-to guide
Is Your Website Blocking AI Crawlers? How to Check
To check whether your website blocks AI crawlers, read your robots.txt for AI bot names, then request a page as each bot and look at the status code. A 200 response and no Disallow rule means access is open. A 403, a 429 or a challenge page means a block.
Check by Hand in Four Steps
You can check by hand in four steps: read robots.txt, request a page as the bot, view the page source and look for noindex signals. Each step answers a different question, so do all four.
Read robots.txt
Open yourdomain.com/robots.txt. Search for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended. A User-agent line followed by Disallow: / blocks that bot. A User-agent: * group with Disallow: / blocks everyone.
Request a page as the bot
From a terminal, run curl -I -A "OAI-SearchBot" https://yourdomain.com/ and read the first line. Use each bot's current user-agent string from its vendor's documentation. A 200 is good. A 403 or 429 means the server refused it.
View the page source
Check that your main text is in the HTML the server sends, not only added by JavaScript afterwards.
Check noindex signals
Look for a robots meta tag or an X-Robots-Tag header with noindex on pages you want found.
A request from your own computer is not the same as a request from the vendor. Some firewalls verify real bots by network address, so a fake user agent can pass or fail differently. Treat a hand test as a strong hint, not final proof.
How to Read the Results
| You see | It means | Next step |
|---|---|---|
| 200 and no Disallow rule | The bot can reach the page | Check that the content is in the HTML |
| Disallow rule in robots.txt | You or a tool told the bot to stay away | Remove or narrow the rule if you want that bot |
| 403 Forbidden | The server or a firewall refused the request | Look at firewall, CDN and security plugin rules |
| 429 Too Many Requests | A rate limit is triggered | Raise the limit for known crawlers |
| A challenge or CAPTCHA page | Bot protection treats the bot as suspicious | See [Cloudflare blocking AI bots](/geo/cloudflare-blocking-ai-bots) if you use Cloudflare |
Check Automatically
The free GeoLint audit does all of this for you. Enter a page URL and it reads robots.txt, requests the page as each AI search bot, loads it with and without JavaScript and reports what is blocking you. No signup is needed. How GeoLint works lists every check.
If you use a hosted platform, the platform guides cover where to change settings: Shopify, WordPress, Wix and Webflow. For the full list of causes, read why ChatGPT cannot see your website.
Checking AI Crawler Access: Common Questions
How do I know if GPTBot is blocked on my site?
Open your robots.txt and look for a GPTBot group with Disallow: /. Then request a page with the bot's user agent and look at the status code. A 403 or 429 means the server blocks it even if robots.txt does not.
Is there a free AI crawler checker?
Yes. The GeoLint audit on this site is free, needs no signup and tests each AI search bot against your page.
Does robots.txt allowing a bot guarantee access?
No. A firewall, CDN or rate limit can refuse the bot even though robots.txt allows it. That is why you test the response as well as the file.
Should I allow every AI crawler?
It is your policy choice. Many owners allow search and user bots and decide separately about training bots. See the [robots.txt guide](/geo/robots-txt-ai-bots).
Check your own website for free
GeoLint asks each AI search bot for your page, loads it with and without JavaScript and shows what is blocking you. It is free, needs no signup and takes about 20 seconds.