Platform guide
Webflow and AI Crawlers: robots.txt Settings and Testing
Webflow lets you write a custom robots.txt in Site Settings under SEO, and block bots by user agent. To be found by AI search, make sure the file does not block the search bots, then test the live site, because a CDN in front of Webflow can block them too.
Where Webflow Controls robots.txt
In Webflow, the custom robots.txt is edited in Site Settings under SEO. Webflow's own blog gives examples such as blocking GPTBot from the whole site, and notes that well-behaved crawlers respect these rules while content already indexed is not removed. Publish the site after saving so the live file changes.
Webflow also has indexing controls for the whole site, for the webflow.io staging subdomain and for individual pages. A page set to noindex or a site where staging indexing is off is a separate issue from AI bot access, so check both.
Allow AI Search Bots in Webflow
To allow AI search, leave out any Disallow rule for the search bots or add explicit Allow groups. Block training bots only if that is your policy.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /For what each bot does, read the GPTBot guide, ClaudeBot and its sister bots and PerplexityBot. Menu names can change, so check Webflow's current documentation.
When Cloudflare Sits in Front of Webflow
If your domain's DNS runs through Cloudflare, Cloudflare's AI bot settings apply to your Webflow site as well. A block there overrides a permissive robots.txt. Webflow has said it is working with Cloudflare on tools for controlling how AI uses content, so the details may change.
Read Cloudflare blocking AI bots for the three places to check.
Test the Published Site
Run the free GeoLint audit on your published domain, not the webflow.io address. It shows whether robots.txt allows each AI search bot and what the server returned, including blocks from a CDN or rate limit.
Webflow and AI Crawlers: Common Questions
Can I edit robots.txt in Webflow?
Yes. Webflow documents a custom robots.txt in Site Settings under SEO. Publish the site after saving. Check Webflow's current documentation if the menu has moved.
Does Webflow block AI crawlers?
Webflow says there were no immediate changes to its sites when Cloudflare announced AI bot controls, and that customers manage crawlers through their own robots.txt. Your own file or a CDN in front of the site may block them, so test it.
Why can AI bots not read my Webflow site even though robots.txt allows them?
A CDN, firewall or rate limit in front of the site may be refusing the request. Run an audit to see the status code the bot receives.
Does a noindex page affect AI search?
It can. Some AI search tools use search indexes, so a page you hide from search may also be hidden from them. Remove noindex from pages you want found.
Check your own website for free
GeoLint asks each AI search bot for your page, loads it with and without JavaScript and shows what is blocking you. It is free, needs no signup and takes about 20 seconds.