Platform guide
WordPress and AI Crawlers: How to Allow or Block Them
WordPress usually serves a virtual robots.txt that it or an SEO plugin generates, so there may be no file to edit. Read yourdomain.com/robots.txt to see the live rules, then check the places that can block AI bots even when robots.txt allows them.
Where Your WordPress robots.txt Comes From
Your robots.txt comes from one of three places: a physical file in the site's root folder, a virtual file that WordPress generates, or one managed by an SEO plugin. A physical file takes priority over a virtual one, and a plugin can overwrite manual edits, so find out which applies before you change anything.
Open yourdomain.com/robots.txt in a browser. That is the file crawlers read. Editing something else, such as a copy that your SEO plugin no longer serves, changes nothing.
What Can Block AI Bots on WordPress
Five places can block an AI bot on a WordPress site, and robots.txt is only the first. Check them in order.
| Where | How it blocks | How to check |
|---|---|---|
| robots.txt | Disallow lines for named bots, or for everyone | Open /robots.txt and read it |
| Search engine visibility setting | The Discourage search engines option in Settings, Reading adds a noindex signal and can change robots.txt. It is meant for sites under construction | Check that it is unticked on a live site |
| SEO or AI-blocking plugin | Some plugins add AI bot rules to robots.txt or headers with one switch | Review the plugin's bot or crawler settings |
| Security plugin or firewall | Rate limits or bot rules can return 403 or 429 to crawlers | Test as the bot to see the status code |
| Host or CDN | Hosting firewalls and CDNs such as Cloudflare can block by user agent | See [Cloudflare blocking AI bots](/geo/cloudflare-blocking-ai-bots) |
How to Allow the AI Bots You Want
Remove any rule that blocks a bot you want, using the same tool that created it. If your robots.txt is served by an SEO plugin, edit it in that plugin's robots.txt editor. If it is a physical file, edit the file through your host's file manager or SFTP.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Keep wp-admin and other private paths blocked for all bots. The robots.txt guide explains how Allow and Disallow groups combine and which bots do what, and the GPTBot guide covers OpenAI's three crawlers.
Verify With a Real Request
After changing anything, run the free GeoLint audit on your homepage. A clean robots.txt is not proof that a security plugin or your host lets the bot through, and the audit shows the real status code for each AI search bot.
If the audit finds a block you cannot locate, or you would rather not edit server rules yourself, see what the done-for-you fix includes.
WordPress and AI Crawlers: Common Questions
Does WordPress block AI bots by default?
WordPress itself does not name AI bots in its default robots.txt. A plugin, a security setting, the Discourage search engines option or your host can still block them, so read your live robots.txt and test the page.
Why is my WordPress robots.txt not changing when I edit it?
Another source is probably serving it. A physical file overrides the virtual one, and an SEO plugin can regenerate its own. Find which one produces the file at /robots.txt and edit that.
Is there a plugin to block AI crawlers?
Yes, several plugins add AI bot rules with one setting. Be careful that they do not also block the search bots you want. Check the resulting robots.txt after enabling one.
Does the Discourage search engines setting block AI bots?
It asks search engines not to index the site, and it can change robots.txt. It is meant for sites under construction. On a live site it should be off if you want to be found by search or AI tools.
Check your own website for free
GeoLint asks each AI search bot for your page, loads it with and without JavaScript and shows what is blocking you. It is free, needs no signup and takes about 20 seconds.