To configure robots txt for ai search, allow retrieval crawlers like OAI-SearchBot and PerplexityBot, but block training scrapers like GPTBot and Google-Extended. This split lets ChatGPT Search and Perplexity index and cite your pages in real time. It also stops AI firms from using your copy to train future foundation models without your consent.
Treating every bot as a threat costs you traffic. Block retrieval agents, and your site drops out of the answers high-intent buyers read before they purchase.
What is the difference between AI training crawlers and search indexers?
AI web bots split into two camps: training scrapers and search indexers. Training scrapers pull text across the web to build foundation models. Search indexers fetch pages in real time to show live links alongside conversational answers.
The split between gptbot vs oai-searchbot shows how this works in practice. OpenAI runs GPTBot solely to train foundation models like GPT-4. Disallowing GPTBot keeps your writing out of training data, but it does not pull your pages from search results. OAI-SearchBot does the opposite job. OpenAI sends it out to index content for ChatGPT Search. Block OAI-SearchBot, and ChatGPT cannot crawl your pages or cite your links in search replies.
OpenAI also uses a third crawler named ChatGPT-User. It runs only when someone pastes a direct link into a prompt, so it follows fetcher rules rather than automated crawl logic.
Perplexity works the same way. Its main bot, PerplexityBot, builds a search index. Perplexity states that PerplexityBot never scrapes pages to train foundation models. It relies on Perplexity-User for links that users submit in chat.
Which AI user-agents should you allow to appear in search citations?
If you want buyers to find your products inside AI answers, open your doors to search bots while blocking model scrapers.
Engines do not pick the same links. A study of 680 million AI citations by 5WPR showed that ChatGPT and Perplexity share only 11% of their sources. Research by Leapd across 34,234 prompt responses found that Perplexity cited brand domains in 13.05% of its answers. ChatGPT cited brand domains in just 0.59% of answers, leaning on broad sources like Wikipedia while Perplexity pulled fresh web pages.
Give each agent explicit rules in your robots.txt file:
| User-Agent | Operator | Function | Recommended Rule | Search Citation Impact |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search retrieval | Allow | High: Disallow removes your site from ChatGPT Search. |
| GPTBot | OpenAI | Foundation model training | Disallow or Allow | None: Search visibility remains intact if blocked. |
| ChatGPT-User | OpenAI | User-initiated link fetch | Allow | Medium: Prevents users from referencing your URLs in chats. |
| PerplexityBot | Perplexity | Retrieval index | Allow | High: Disallow removes your pages from Perplexity citations. |
| Google-Extended | Gemini training & Vertex AI | Disallow or Allow | None: Does not impact Google Search or AI Overviews. |
Setting perplexitybot robots txt access to Allow keeps you visible in an engine that links to commercial websites at a high rate.
Can you opt out of Google AI Overviews without losing organic rankings?
You cannot use robots.txt to opt out of google ai overviews and keep your regular organic ranks at the same time.
Googlebot powers both traditional search results and AI Overviews. Google offers a separate token called Google-Extended, but that only controls Gemini model training and Vertex AI. Blocking Google-Extended will not stop your pages from showing up in AI Overviews or standard search snippets.
Google states that the only way to leave AI Overviews is through snippet directives: the nosnippet, data-nosnippet, or max-snippet meta tags. Use those, and Google strips descriptions from your classic organic listings too. Losing your snippets hurts click rates across standard search results.
This forces a hard choice. A Search Engine Land study of 53 brands and 5.47 million queries found that cited links in AI Overviews earned a 2.1% CTR. Uncited organic listings on that same page sank to 0.9%. The study noted that AI Overviews showed up on 95% of comparison queries and 86% of question queries, but only 5% of transactional product searches.
Rules also shift by region. Google AI Overviews run in over 200 countries, yet antitrust action in the UK and Digital Markets Act rules in the European Union force different layouts than what users see in the US.
Why do Web Application Firewalls silently block AI bots with 403 errors?
Your robots.txt file can welcome an AI bot while your network edge shuts the door. Firewalls and Content Delivery Networks frequently drop AI traffic before the crawler ever sees your permissions.
Cloudflare, AWS WAF, and Akamai run automated bot filters. Cloudflare includes an "AI Crawl Control" switch made to stop site scrapers. Turn it on without custom rules, and the firewall checks incoming traffic at the edge. If a crawler introduces itself as OAI-SearchBot or PerplexityBot, the server sends back an HTTP 403 Forbidden code right away.
Your robots.txt file sits on the origin server. When an edge firewall drops the connection, the crawler never gets to check your rules. It simply marks your site down as broken and removes your pages from its index.
Check your security logs to fix this. Look for 403 errors matched to AI user-agents. Whitelist verified search indexers with ASN and reverse DNS checks instead of user-agent strings alone, since scrapers can easily fake a header name.
What happens to ChatGPT Search when your robots.txt uses wildcard disallows?
Problems start when site owners try to block ai crawlers robots txt rules with broad wildcard statements.
Consider this standard block:
``txt User-agent: * Disallow: / ``
OpenAI bots take that star rule literally. If you do not write a direct Allow rule for OAI-SearchBot, ChatGPT Search treats your entire domain as off-limits.
This happens all over the web. Angeio audited 762 Magento 2 storefronts and discovered that 14% had cut off answer-engine crawlers by mistake. On top of that, 13% of stores blocked retrieval bots while trying to shut out training bots.
To keep ChatGPT Search running while blocking training bots, list each user-agent on its own line:
```txt User-agent: OAI-SearchBot Allow: /
User-agent: GPTBot Disallow: / ```
Without this direct pair, wildcards will hide your product catalogs and comparison articles from live conversational queries.
Does blocking GPTBot and foundation scrapers hurt your brand long term?
Blocking GPTBot keeps OpenAI from training on your text, but the move carries side effects for brand awareness.
Foundation models write answers based on facts absorbed during pre-training. Block GPTBot, and the model never reads your spec sheets, product docs, or pricing pages. When later model versions answer category questions without browsing the live web, they must lean on whatever text they have: forum threads, Reddit rants, and competitor blogs.
If your rivals allow GPTBot, their feature lists and market claims settle into the model's memory. Your brand gets described through second-hand takes or left out of offline answers.
Depth and proof still drive citations when engines do browse live. Princeton and IIT Delhi researchers found that adding original statistics to a page lifted AI citation rates by 41%. Adding direct quotes from experts drove citations up by 40%. Blocking foundation crawlers stops models from absorbing those source signals into their base weights.
Frequently asked questions
What is the exact technical difference between GPTBot and OAI-SearchBot?
GPTBot crawls pages to train OpenAI's base language models. OAI-SearchBot indexes web pages for live ChatGPT search results. Blocking GPTBot does not hurt your search presence, but blocking OAI-SearchBot removes your site from ChatGPT Search links.
Does an llms.txt file help AI crawlers index my site better?
No search engines or AI bots treat llms.txt as a crawling or ranking signal. Googlebot, PerplexityBot, and OAI-SearchBot discover and read content through plain HTML, XML sitemaps, and robots.txt files.
Why do AI Overviews deep link users into the middle of my web pages?
AI engines use Chrome Text Fragment URLs (#:~:text=) to send users straight to the sentence cited in their answer. Keep your key buttons, navigation links, and proof points visible so users who land mid-page can still browse.
Do AI search engines respect robots.txt when a user enters a direct URL into a prompt?
When a user pastes a specific URL into a prompt, OpenAI calls ChatGPT-User instead of OAI-SearchBot. AI firms honor robots.txt for automated indexing runs, but live user-directed URL fetches follow separate fetcher rules.
Recommended Robots.txt Strategy for AI Search
Managing AI bots takes precision, not a single master switch. Allow retrieval agents like OAI-SearchBot and PerplexityBot so conversational search engines can cite your pages. If you want to protect your text from future training sets, disallow GPTBot and Google-Extended without losing your search rankings.
Audit your firewall. Check Cloudflare or AWS WAF logs to confirm edge rules are not throwing 403 errors at real search crawlers before your robots.txt file can speak for itself.
Sources
- OpenAI Crawlers and User-Agents Documentation — Technical distinction between GPTBot for training and OAI-SearchBot for search indexing.
- OpenAI Documentation on Bot Handling and Wildcards — Details on ChatGPT-User fetching logic and wildcard disallow behavior.
- PerplexityBot Web Crawler Documentation — Confirmation that PerplexityBot indexes for search without training foundation models, and notes on WAF edge blocks.
- Google Search Central Overview of Google Crawlers — Role of Google-Extended in Gemini training and confirmation it does not impact Search rankings.
- Google Search Central: Controlling AI Features — Requirements for opting out of AI Overviews using nosnippet tags rather than robots.txt.
- Audit of AI Crawler Directives on E-Commerce Platforms — Statistics showing 14% of audited stores blocked answer engines and 13% accidentally blocked retrieval bots.
- AI Overviews CTR and Query Type Analysis — CTR metrics for cited vs non-cited listings and query distribution data.