Block AI Crawlers in robots.txt and Stay in Google Search

To block AI crawlers in robots.txt and stay in search, disallow the training tokens GPTBot and Google-Extended, leave OAI-SearchBot and Googlebot allowed, and check the live file today. Blocking Google-Extended does not remove your site from Google Search, and Google’s crawler documentation (opens in a new tab) says so directly.

Each AI company publishes separate tokens for training, search and user-triggered fetching. If you paste a generic "block all AI" list, you can drop out of ChatGPT search results while believing you only opted out of training. The fix takes one file and about ten minutes.

Key takeaways

  • Disallow GPTBot and Google-Extended to opt out of training, and leave OAI-SearchBot, ChatGPT-User and Googlebot allowed to stay findable.
  • Google-Extended has no separate HTTP user agent string, so it never shows up when you search your logs for it.
  • Run curl against /robots.txt first, because a physical file, WordPress or Cloudflare may be serving it.
  • robots.txt compliance is voluntary, so Cloudflare’s AI Crawl Control is the option that enforces a block.

Block GPTBot and Google-Extended, keep OAI-SearchBot and Googlebot

Disallow GPTBot and Google-Extended and you opt out of model training while staying in ChatGPT search and Google Search. Google states that Google-Extended does not impact a site’s inclusion in Google Search and is not used as a ranking signal (opens in a new tab). Split every bot by the job it does, then decide per job.

JobUser agent tokenOperatorWhat a Disallow does
TrainingGPTBotOpenAIOpts the site out of OpenAI model training
Training and groundingGoogle-ExtendedGoogleControls use for Gemini training and grounding, no effect on Google Search
Search and citationOAI-SearchBotOpenAIKeeps pages out of ChatGPT search results
Search and citationGooglebotGoogleRemoves pages from Google Search and related products
User-triggered fetcherChatGPT-UserOpenAIStops page loads that a person requests inside ChatGPT

The OpenAI roles come from its crawler overview (opens in a new tab). One detail trips people up: Google-Extended covers more than training. Google says it also governs grounding, meaning content from the Google Search index supplied to Gemini Apps and Grounding with Google Search on Vertex AI at prompt time.

Warning: Never disallow Googlebot to "stop Google AI". Googlebot preferences affect Google Search, Discover, Images, Video and News. Google-Extended is the only token in this list built to opt out of Gemini training without touching search.

Open yourdomain.com/robots.txt and find out who serves it

Load the live file before you edit anything, because three different systems can serve it. It may be a physical file in the web root, a virtual file generated by WordPress, or a file with Cloudflare directives prepended. Edit the wrong one and nothing on your live site changes.

Bash
curl -s https://yourdomain.com/robots.txt
curl -sI https://yourdomain.com/robots.txt

Read the output in this order:

  1. Look for the line # BEGIN Cloudflare Managed content. If it is there, Cloudflare is prepending directives because Bot Preference Sync is on and at least one policy is set to block. Change the policies in the dashboard instead of the file.
  2. Open the web root (usually public_html) in your file manager or over SFTP. A physical robots.txt there is served directly, and WordPress never generates one.
  3. With no physical file, WordPress core serves a virtual robots.txt. SEO plugins such as Yoast SEO and Rank Math can edit it from their settings.

In our work, the most common expensive mistake is choosing a plugin without considering compatibility with other plugins and the hosting environment. For robots.txt, that looks like an AI-blocker plugin writing a virtual file while a forgotten physical file in the web root silently replaces it. Pick one place to manage the file and delete the others.

The deciding factor is whether you want to appear in AI search answers and citations or want out of AI products entirely. For citations in ChatGPT search and Google, block training only. To leave AI products altogether, block every AI bot. For maximum reach, allow everything.

Your goalPolicyDisallowKeep allowed
Cited in AI search, no trainingTraining onlyGPTBot, Google-ExtendedOAI-SearchBot, ChatGPT-User, Googlebot
Out of AI productsAll AITraining, search and fetcher tokensGooglebot
Maximum reachAllow allNothingEverything

I recommend training only for most content sites. It costs you nothing in search visibility, and it states a clear preference on how your articles are used for training.

Blocking has a trade-off at the market level. A 2026 model by Miyazaki (opens in a new tab) [1] starts from the observation that major newspapers block AI crawlers while parties and partisan sites leave their text open, and asks what readers hear while that hold-out lasts. It is a game-theoretic model and measures nothing about your site, but it is a fair reminder that a block removes your voice from what models read.

Robots.txt governs future crawling only. It does not pull back any data a crawler already collected.

Copy one of three robots.txt files to block AI crawlers or allow them

Paste one of these into the file you identified above. Each uses explicit groups for the named agents, because a crawler follows the group that matches its own token and ignores the wildcard group. Keep any existing rules you need, such as your WordPress admin paths.

Block training only (recommended)

Txt
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.com/sitemap.xml

Block all AI, keep Googlebot

Txt
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Google-Extended
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: CCBot
User-agent: ClaudeBot
User-agent: meta-externalagent
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.com/sitemap.xml

The extra names in the second file match the AI crawlers on Cloudflare’s managed list. This file removes you from ChatGPT search results, and Google-Extended also stops grounding in Gemini Apps.

Allow everything

Txt
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Note: I did not run these files against live crawlers for this article. The syntax and token names follow the official docs as of their last updates (Google’s page: 2026-10-30; Cloudflare’s page: Nov 2, 2026). Google also says it intends to support the AIPREF training signal (user-agent: _usageai-training) in early 2027, so recheck its crawler list then.

Use Cloudflare’s managed robots.txt if the site is already behind it

Cloudflare can write and serve the AI-bot rules for you. Turn on Enable Bot Preference Sync, set Training to Disallow, and Cloudflare prepends managed directives to your robots.txt within a few seconds. The managed robots.txt setting (opens in a new tab) is available on all plans.

To add managed directives for AI training crawlers:

  1. In the Cloudflare dashboard, open Security Settings and filter by Bot traffic.
  2. Select AI Crawler and Agent access policies.
  3. For Training, select Disallow.
  4. Turn on Enable Bot Preference Sync.
  5. Select Save, then reload /robots.txt. No cache purge is needed.

The managed block lists Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. It also adds a Content-signal: search=yes, ai-train=no, use=reference line for all other agents. OAI-SearchBot and Googlebot are not on that list, so the Training policy matches the training-only file above.

The Search and Agent policies add their own directives when set to Block, so read the served file after every policy change. If your origin has a robots.txt that returns HTTP 200, Cloudflare puts its block in front of it and serves one combined response. Without an origin file, Cloudflare creates one.

There are two caveats from Cloudflare’s docs. Google Search Console may report "Syntax not understood" for Content Signals, and Cloudflare says it has seen no effect on crawling or SEO from those reports. Cloudflare also states that robots.txt compliance is voluntary, so you need AI Crawl Control when you want an enforced block.

Verify the live file and your logs today

Reload the live file, confirm the groups you intended, then search your access logs for the AI user agents. Do it the same day you edit, because a plugin, a physical file or a Cloudflare policy can quietly change what is served.

Bash
curl -s https://yourdomain.com/robots.txt | grep -iE -A1 'GPTBot|Google-Extended|OAI-SearchBot|Googlebot'
grep -iE 'GPTBot|OAI-SearchBot|ChatGPT-User' /var/log/nginx/access.log | tail -n 20

Adjust the log path for your server. Google-Extended will not appear in the second command’s output. Google says (opens in a new tab) it has no separate HTTP user agent string and is used only as a control token, so the crawling itself arrives under existing Google user agents.

User agent strings can be spoofed. For Googlebot, Google documents a reverse DNS check, where the hostname matches crawl-–––.googlebot.com or geo-crawl-–––.geo.googlebot.com.

Next step: open the OpenAI crawler overview (opens in a new tab) and Google’s crawler list, compare the token names with your live file, and set a reminder to recheck both when the AIPREF support Google plans for early 2027 ships.

Frequently Asked Questions

How do I block AI crawler bots with robots.txt?

Add a User-agent group for each bot token, followed by Disallow: /. To opt out of training only, that means GPTBot and Google-Extended. Edit the file your site actually serves, then confirm the change with curl against /robots.txt.

Does blocking AI crawlers affect Google Search?

Blocking Google-Extended does not. Google states it does not impact a site’s inclusion in Google Search and is not used as a ranking signal, per its crawler documentation (opens in a new tab). Disallowing Googlebot is the rule that removes pages from Google Search.

Will blocking GPTBot remove my site from ChatGPT search?

No. OpenAI lists GPTBot as the training crawler and OAI-SearchBot as the one for search, per its crawler overview (opens in a new tab). Disallow GPTBot to opt out of training, and leave OAI-SearchBot allowed to stay in ChatGPT search results.

Does robots.txt actually stop AI crawlers?

It states your preference, and compliant crawlers follow it. Cloudflare says robots.txt compliance is voluntary (opens in a new tab), so use Cloudflare’s AI Crawl Control when you need an enforced block. A robots.txt change also does not remove data a crawler already collected.

Why does Google-Extended never appear in my server logs?

Google-Extended has no separate HTTP user agent string. Google uses it only as a control token in robots.txt, and the crawling itself arrives under existing Google user agents. Search your logs for GPTBot, OAI-SearchBot and ChatGPT-User instead.

Where do I edit robots.txt on a WordPress site?

It depends on what serves the file. A physical robots.txt in the web root wins, otherwise WordPress serves a virtual one that Yoast SEO or Rank Math can edit. If the site is behind Cloudflare with Bot Preference Sync on, change the policy in the dashboard.

References

  1. "Newspapers Blocking AI Crawlers Expose Their Voters to a Partisan Flood", Miyazaki, 2026. doi:10.31235/osf.io/8m6he_v2 (opens in a new tab)