AI crawler guide
Cloudflare Blocking AI Bots: How to Check and Fix It
Cloudflare can keep AI crawlers away from your website in three places: its AI bot policies for Search, Agent and Training bots, a managed robots.txt that disallows AI crawlers, and per-crawler blocks or bot rules. Check all three, because fixing only robots.txt leaves a firewall block in place.
Why Cloudflare Can Block AI Bots
Cloudflare sits in front of your website and decides which requests reach it, so a rule there can stop an AI crawler even when your own robots.txt allows it. Some of these controls can be switched on without you noticing, for example by a developer, at setup, or by a default.
Cloudflare announced in July 2025 that it would block AI crawlers by default for the websites it hosts, and let owners allow or block bots case by case. Its July 2026 documentation says that from 15 September 2026 newly added domains get updated defaults: Training and Agent bots are blocked on pages that show ads, while Search bots stay allowed. An older website depends on the settings chosen earlier, and Cloudflare's defaults change over time, so check your own dashboard.
The Three Places to Check
Look at the AI bot policies, the robots.txt your site actually serves, and the per-crawler blocks and rules. Any one of them can block a bot you want, and fixing one does not undo the others.
| Where | What it does | What to check |
|---|---|---|
| AI bot policies: Search, Agent and Training (Security Settings, filtered by Bot traffic) | Each behaviour can be allowed, blocked, or blocked only on pages that show ads. Search covers crawlers that index content to answer questions, Agent covers fetches made in real time for a person, and Training covers collecting content to train models | Whether Search and Agent are blocked. Training is the one most owners choose to block |
| Managed robots.txt (Bot Preference Sync) | Adds Disallow lines for AI crawlers, such as GPTBot and ClaudeBot, to the robots.txt your site serves, in front of your own file | Open /robots.txt and look for AI bot lines you did not write |
| AI Crawl Control and custom rules | Sets Allow or Block for each crawler. A block creates a firewall rule that enforces it | Whether a crawler you want is set to Block, and whether any custom rule challenges AI user agents |
These names follow Cloudflare's documentation in October 2026 and can change. Older accounts may still show the earlier single Block AI bots setting, which Cloudflare's documentation said was being retired, so use its current documentation to find the right screens.
How to Check What Cloudflare Is Doing
Check in three steps: read your live robots.txt, review Cloudflare's AI bot policies and crawler table, then test as the bot. The last step is the only proof of what a crawler actually receives.
Read the live robots.txt
Open yourdomain.com/robots.txt in a browser. If it contains Disallow lines for AI bots that you never wrote, a managed robots.txt is probably adding them in front of your own file.
Review Cloudflare's AI settings
Open Security Settings, filter by Bot traffic and open the AI bot policies. Then open AI Crawl Control and check the crawler table for any crawler you want that is set to Block.
Test as the bot
Run the free GeoLint audit. It requests your page as each AI search bot and shows what robots.txt says and what the server really returned, such as a 403, a 429 or a challenge page.
How to Allow the Bots You Want
To allow the bots you want, set the Search policy to allow, set Agent to allow if you want people to open your pages through AI assistants, and keep Training blocked if that is your policy. Then check that nothing in robots.txt or AI Crawl Control still blocks the specific crawlers.
- In the AI bot policies, allow Search, and allow Agent if you want AI assistants to be able to open your pages for people. Block Training if you do not want your content used to train models.
- If Bot Preference Sync is on, remember that it writes the matching Disallow lines into your robots.txt, so a Search or Agent block shows up there too.
- In AI Crawl Control, set Allow on the specific crawlers you want and check for custom rules that challenge AI user agents.
A user agent name can be faked by anyone. Do not create a rule that skips all protection for any request that merely claims to be GPTBot or ClaudeBot. Use the per-crawler controls Cloudflare provides, which are built on its own bot verification.
To decide which bots to allow, read GPTBot, OAI-SearchBot and ChatGPT-User, ClaudeBot and its sister bots, PerplexityBot and robots.txt for AI bots. Cloudflare decides how each bot is classified, so check its bot directory for the policy that covers a given crawler.
Re-test After the Change
After changing a setting, wait a few minutes and run the audit again, because caching and rule updates are not always instant. A pass in GeoLint is a strong sign that your rules are right.
GeoLint cannot fully copy a firewall that verifies a bot by its network address, which is how Cloudflare recognises verified bots. For certainty, also check the AI Crawl Control view and Cloudflare's security events for real crawler requests and whether they were allowed.
Cloudflare and AI Bots: Common Questions
Does Cloudflare block AI bots by default?
Cloudflare announced in July 2025 that it would block AI crawlers by default for the websites it hosts. Its documentation says newly added domains get updated defaults from 15 September 2026, with Training and Agent bots blocked on pages that show ads and Search allowed. For an older website the result depends on the settings that were chosen, so check your dashboard.
Why does my robots.txt contain lines I did not write?
A managed robots.txt can add Disallow lines for AI crawlers in front of your own file. Cloudflare says it does this when you sync your AI bot preferences. Look for it in your AI bot policies, then decide which bots you want to keep blocking.
Will allowing AI bots expose my private pages?
Allowing a bot lets it read public pages it can reach, the same as any visitor. It does not unlock pages behind a login. If you do not want content used for training, allow Search and keep Training blocked.
How do I know the problem is fixed?
Run GeoLint on your page and check that each AI search bot is allowed in robots.txt and receives a normal response from the server. Then look at the AI Crawl Control view in Cloudflare over the following days for real crawler requests.
Check your own website for free
GeoLint asks each AI search bot for your page, loads it with and without JavaScript and shows what is blocking you. It is free, needs no signup and takes about 20 seconds.