Block AI training without losing AI search: robots.txt rules

Training and search are separate decisions, and robots.txt handles them separately. OpenAI documents that GPTBot and OAI-SearchBot are controlled independently, so you can allow search while disallowing training. Blocking GPTBot does not by itself remove a site from ChatGPT search, and robots.txt does not reliably control user-triggered fetchers such as Perplexity-User.

• Decide training and search separately. GPTBot is the training control. OAI-SearchBot controls ChatGPT search.
• Know what a search block costs. Opting out of OAI-SearchBot keeps your site out of ChatGPT search answers. Disallowing GPTBot alone does not.
• Wait before judging a change. OpenAI says search-related updates can take about 24 hours to take effect.
• Treat robots.txt as a request. Check your firewall and CDN, then confirm with server logs.

Why are training and search separate decisions?

Many company policies treat "AI crawlers" as one group. They are not. OpenAI says that with its bots, each setting is independent of the others, so a rule for one does not settle the other.

OpenAI describes the GPTBot rule this way:

"Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models." (OpenAI crawler docs)

Nothing in that sentence says the page disappears from AI search. Start by writing down what you want: keep content out of training, appear in AI answers, or both. Each goal needs different lines in the file, and a policy that mixes them up tends to block more than intended.

Which bot does what?

The names look alike, but each bot has a narrow job.

Bot	What it does	What robots.txt controls
GPTBot	Collects content for OpenAI's training	A Disallow signals content should not be used for training
OAI-SearchBot	Powers ChatGPT search	Sites opted out will not be shown in ChatGPT search answers, though they can still appear as navigational links, per OpenAI
ChatGPT-User	Fetches pages when a user asks	OpenAI says robots.txt rules may not apply, and this agent is not used to decide whether content appears in Search
PerplexityBot	Surfaces and links sites in Perplexity search	Cloute's guide, quoting Perplexity, says it is not used to crawl content for AI foundation models
Perplexity-User	Fetches pages a user requested	Perplexity's crawler documentation says it generally ignores robots.txt
Google-Extended	A robots.txt token, not a crawler	GeoToolbox says it governs use of content for Gemini training and grounding, and does not control Google Search

For a longer rundown of each crawler, see Citable's guide to robots.txt and AI crawlers.

Does blocking GPTBot remove you from ChatGPT search?

No. A GPTBot block addresses training. The line that decides whether you appear in ChatGPT search answers is the OAI-SearchBot rule. A guide on GPTBot and WAF settings calls the opposite assumption a common misunderstanding in enterprise policies.

A split policy looks like this:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

This keeps content out of OpenAI's training while leaving the site eligible for ChatGPT search. Change the OAI-SearchBot line only if you want to leave search results.

If you want ChatGPT to recommend your business, check the OAI-SearchBot rule before anything else. After you edit it, OpenAI says its systems need roughly 24 hours to adjust for search results.

What can robots.txt control at Perplexity?

PerplexityBot is the search crawler. According to Cloute's guide quoting Perplexity, the company recommends allowing it so your site can appear in its search results. Blocking it targets exactly that visibility.

Perplexity-User is different. Because a person requested the fetch, Perplexity's documentation says this fetcher generally ignores robots.txt rules. A Disallow for it is not a reliable control.

For a firm block, Perplexity's documentation says to use the current IP ranges from its official JSON endpoints as the source of truth for WAF configurations. Build firewall rules from that list. User-agent strings alone do not tell you who is asking.

Compliance has also been questioned. One guide to PerplexityBot reports that Perplexity was accused of using undeclared crawlers in 2024 and 2025. If keeping Perplexity out matters to you, add a network-level rule on top of the file.

Does a robots.txt rule actually stop the bot?

Robots.txt is a set of instructions your server publishes. Whether a bot receives a page also depends on the server, the CDN and any firewall in front of it. The Murat Ulusoy guide notes that many domains allow GPTBot in robots.txt but block it through Cloudflare, Akamai or AWS WAF, and the file itself does not reveal that.

Two more limits apply:

• Timing: a block applies from the next fetch onward. The same guide says it is not retroactive for models already trained.
• Popularity: TechnologyChecker's analysis of robots.txt files across Cloudflare's network found GPTBot the most-named AI user agent in Disallow rules, with publishers blocking training bots more than answer bots.

An allow line tells you what you intended. It does not tell you what happens.

How do you test your rules?

Testing takes a few minutes and catches most gaps between policy and behavior.

• Read the live file: open /robots.txt on the live domain and confirm it serves the rules you wrote.
• Run a checker: Citable's AI crawler checker shows whether robots.txt blocks ChatGPT, Perplexity or Claude.
• Check the logs:
• Wait for search changes: give search-related edits about a day before you decide they failed.
• Recheck after infrastructure changes: a new firewall, CDN or bot-management update can override a robots.txt allow.

Which setup fits your business?

• Publishers with paid content: consider disallowing the training bots while keeping search bots allowed, if that matches your content policy.
• Local businesses that want AI recommendations: the search bots matter more than the training bot. Disallowing GPTBot does not remove you from ChatGPT search.
• Ecommerce brands: test product and category pages after any change. A WAF or CDN rule can block a bot that robots.txt allows, and product pages are where that costs you most.
• Agencies: document each client's decision and the reason, and recheck it after hosting changes.

A Scalableloops survey found that none of 22 small and mid-sized business websites blocked the search crawlers of ChatGPT, Claude or Perplexity in robots.txt. Whatever you choose, make it a decision rather than a default.

Next steps

• Write one sentence stating your goal: opt out of training, appear in AI answers, or both.
• Open your live /robots.txt and find the GPTBot, OAI-SearchBot and PerplexityBot lines. Add the ones that are missing.
• Review your firewall and CDN bot rules, and compare any Perplexity rules with its published IP ranges.
• Check your server logs a day after the change for the bot names you expect.
• If you want a starting file, Citable's robots.txt generator lets you allow or block GPTBot, ClaudeBot and PerplexityBot, and you can edit the result.

Frequently asked questions

Should I block GPTBot if I want ChatGPT to recommend my business?

You don't need to. GPTBot is the training control, and OAI-SearchBot decides search visibility. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, so keep that bot allowed.

Does blocking GPTBot remove my site from ChatGPT search results?

Not by itself. OpenAI describes a GPTBot Disallow as a signal about training, not as a block on search visibility. Opting out of OAI-SearchBot is what removes a site from ChatGPT search answers.

Will blocking PerplexityBot stop Perplexity from citing my pages?

PerplexityBot is the crawler that surfaces and links sites in Perplexity search, according to Cloute's guide quoting Perplexity. Blocking it works against that visibility, and Perplexity recommends allowing it. Perplexity-User is a separate fetcher with different behavior.

Does Perplexity-User respect robots.txt?

Generally not. Perplexity's documentation says that because a user requested the fetch, this fetcher generally ignores robots.txt rules. Use firewall rules based on Perplexity's published IP ranges if you need a firm block.

How long does it take for a robots.txt change to affect AI crawlers?

For ChatGPT search, OpenAI says it can take around 24 hours from a robots.txt update for its systems to adjust. A block applies from the next fetch onward, according to the Murat Ulusoy guide, and does not undo training that already happened.

Sources

• Overview of OpenAI Crawlers
• Perplexity Crawlers - Perplexity
• What is PerplexityBot? Perplexity's crawlers explained · Cloute
• AI Crawlers: The List of AI Bots & How to Control Them
• GPTBot: OpenAI's Crawler - robots.txt & WAF · Murat Ulusoy
• We Analyzed robots.txt Across Cloudflare's Network (TechnologyChecker.io)
• What Is PerplexityBot? The AI Crawler Behind Perplexity AI - Sunil Pratap Singh
• AI Crawlers in robots.txt: Managing GPTBot and Co. · Scalableloops