Skip to content
MMeridian Digital

How to Audit AI Crawler Access in 5 Steps

Audit AI crawler access in 5 steps: check robots.txt, test with curl, catch hidden Cloudflare blocks, and confirm ChatGPT and Perplexity can see you.

Andy Brians (Claude)
  • robots.txt AI bots
  • ChatGPT crawler
  • Perplexity bot access
  • Google-Extended
  • AI search crawl audit
Checking robots.txt is step one in auditing AI crawler access.
Checking robots.txt is step one in auditing AI crawler access.

To audit AI crawler access, check five things in order: your robots.txt file, a live crawl test with curl, your Cloudflare or WAF rules, a real prompt in each AI engine, and a fix-and-retest step. Do all five. A block at any one layer can make every other AI search fix useless.

Skip a step and you might spend hours polishing content a bot never reaches. This guide walks through all five checks, including the Cloudflare and WAF blocks that robots.txt alone can't show you.

Step 1: Check your robots.txt for AI crawler declarations

Go to yoursite.com/robots.txt in your browser. AI crawlers read this file first. It should name each bot, not just carry one blanket "User-agent: *" rule.

Look for these agents, one at a time:

  • GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI)
  • ClaudeBot, Claude-SearchBot, Claude-User (Anthropic)
  • PerplexityBot, Perplexity-User (Perplexity)
  • Google-Extended (Google's AI training and grounding)

Each one needs its own "Allow: /" line, or no "Disallow" line at all, if you want that engine to crawl your site (Anagram AI, June 2026). A single wildcard block shuts out every AI bot at once. That can happen even if you never meant to touch ChatGPT.

Before, blocking everything: User-agent: * Disallow: /

After, opening the site to named AI crawlers: User-agent: GPTBot Allow: /

User-agent: PerplexityBot Allow: /

User-agent: ClaudeBot Allow: /

User-agent: Google-Extended Allow: /

This check takes five minutes. It catches the most common mistake: a robots.txt written years ago as a CMS default, still quietly blocking every AI engine today.

A robots.txt file with each AI crawler declared and allowed by name.
A robots.txt file with each AI crawler declared and allowed by name.

Step 2: Test crawler access with a command-line tool

robots.txt only states intent. A live request shows what actually happens. Open a terminal. It's built into Mac and Linux; Windows users can use WSL or Git Bash. Run curl with each bot's user-agent string.

Try this for each engine:

  • curl -A "GPTBot" -I https://yoursite.com/
  • curl -A "PerplexityBot" -I https://yoursite.com/
  • curl -A "ClaudeBot" -I https://yoursite.com/
  • curl -A "Google-Extended" -I https://yoursite.com/

Check the first line of the response. A 200 means the page loaded. A 403 or 429 means something blocked the request before robots.txt logic ran. Usually that's a firewall or a bot-management setting. A CAPTCHA page or a redirect loop counts as a fail too, even with no clean 4xx code.

Run the same test on two or three pages, not just the homepage. Try a blog post, a product page, and a page buried a few clicks deep. Sites often guard the homepage well and forget the rest. OpenAI's crawlers respect robots.txt and never try to dodge a block (Cloudflare, July 2026). So a clean pass here is a fair sign your OpenAI-facing pages are open. Perplexity is a different story, covered next.

A live curl test confirms whether GPTBot actually gets past the firewall, not just past robots.txt.
A live curl test confirms whether GPTBot actually gets past the firewall, not just past robots.txt.

Step 3: Verify Cloudflare and WAF aren't blocking AI user-agents

robots.txt and a curl test can both look fine. Yet your site can still be invisible to an AI engine. A firewall in front of your server makes its own call before robots.txt ever gets read. Cloudflare's Bot Fight Mode, a WAF rule set, or a security plugin can quietly serve a challenge page or a flat block to a bot it doesn't recognize.

Check three places:

  1. Cloudflare dashboard, under Security then Bots. Confirm GPTBot, ClaudeBot, and PerplexityBot show as "Allowed," not "Blocked" or "Managed Challenge."
  2. WAF event log, filtered to the last seven days, searched for each bot name. Any "Block" or "Challenge" action next to an AI user-agent is the problem.
  3. Rate-limit rules. A rule built to stop scraping can also catch a real AI crawler that hits many pages fast.

Here's the catch: Cloudflare's own security team found Perplexity using undeclared crawlers. These used generic browser user-agents and rotating IPs to keep reading pages after its named bots, PerplexityBot and Perplexity-User, were blocked (Cloudflare, July 2026). So a clean robots.txt and a passed curl test for PerplexityBot don't guarantee Perplexity stays out, or in, whichever you want. Fully blocking Perplexity takes IP and behavior rules too, not robots.txt alone.

Step 4: Query each engine with a test prompt to see if you appear

The clearest proof crawler access works is an answer that cites you. Open ChatGPT, Perplexity, Claude, Gemini, and Copilot in separate tabs. Ask each the same question your buyer would ask, one your page actually answers.

Note three things per engine: does your brand show up at all, is the citation a clickable link, and how does the wording compare to your own page. Perplexity links almost every claim to a source. That makes it the easiest engine to check (Trustmary, May 2026). ChatGPT is harder to judge. Only about 20% of its mentions carry a clickable citation visible in analytics. So a plain mention with no link still counts, even if GA4 never sees it (getpassionfruit, 2026).

Claude is a special case. Its web search runs on Brave Search. Independent checks found roughly 87% overlap between Claude's cited sources and Brave's top organic results (mostailabs, May 2026). If Brave hasn't indexed your page, Claude likely won't cite it, no matter how open your robots.txt is.

Run three to five prompts per engine as a baseline, and repeat monthly. One firm argues you need 60 to 100 runs per prompt for full statistical confidence. Treat that as a stress test for teams with the tooling to run it, not a rule everyone must follow (getpassionfruit, 2026).

Step 5: Fix access issues and retest within 48 hours

Once you know where the block sits, fix it at that layer. A robots.txt problem needs a robots.txt edit. A firewall problem needs a change to a bot-management setting. No amount of robots.txt tweaking will fix that instead.

Common fixes, in order of how often they show up:

  1. Add missing Allow lines for each AI user-agent in robots.txt.
  2. Move any AI bot from Block or Managed Challenge to Allow in Cloudflare or your WAF.
  3. Whitelist AI bot IP ranges where your provider publishes them, instead of relying on user-agent strings alone.
  4. Remove rate-limit rules that catch fast, repeated requests from one legitimate bot.
  5. Clear any cache layer that might still serve an old, blocking version of robots.txt.

After each fix, retest with the same curl commands from step 2. Then wait 48 hours and run your test prompts again. Two days gives crawlers time to revisit and gives engines time to refresh their index. Test five minutes after a change and you'll usually just see the old, cached result. This step matters more than any content or schema fix on the page. A blocked crawler makes every other optimization pointless (everydayonai, June 2026).

Frequently asked questions

How do I audit whether AI engines can actually crawl my site?

Follow the five checks in order: read robots.txt, run curl with each bot's user-agent, check your Cloudflare or WAF bot rules, ask each engine a real prompt, then fix and retest within 48 hours. Skipping the firewall check is the most common mistake. It's the block robots.txt alone can't show.

Should I block or allow Google-Extended in my robots.txt?

Allow it if you want your pages considered for Google's AI Overviews, AI Mode, and Gemini grounding. Google-Extended controls exactly that, separate from regular Googlebot indexing. Block it only if you've decided the trade-off against AI training use isn't worth the citation potential for your site.

Why does my site rank #1 in Google but get zero citations in ChatGPT?

ChatGPT search draws mainly from Bing, not Google's index. So a top Google ranking says nothing about your Bing visibility. Check Bing Webmaster Tools separately. Confirm OAI-SearchBot and ChatGPT-User aren't blocked at the firewall layer even if robots.txt looks clean.

Does Brave Search matter if I don't care about Claude?

Yes, if any of your buyers use Claude's web search. It runs on Brave's index with about 87% source overlap (mostailabs, May 2026). If Brave hasn't crawled or indexed your page, Claude has almost nothing to cite, no matter your robots.txt settings.

Start With the Firewall, Not the Content

Most AI visibility advice starts with content structure, headings, and schema. Start one layer earlier: confirm the bots can even reach the page. A single Cloudflare rule or a stale robots.txt file can undo months of content work in one line.

Run this checklist once, then again after any hosting, CDN, or firewall change:

  • robots.txt lists GPTBot, ClaudeBot, PerplexityBot, and Google-Extended by name, each with an Allow rule
  • curl with each bot's user-agent returns a 200, not a 403, 429, or challenge page, on the homepage and two deeper pages
  • Cloudflare or WAF bot dashboard shows each AI bot as Allowed, not Blocked or Managed Challenge
  • a real test prompt in ChatGPT, Perplexity, Claude, Gemini, and Copilot returns at least a plain mention, even without a clickable link
  • every fix gets a retest 48 hours later, not five minutes later

Do this audit every quarter, or right after any change to your hosting, CDN, or security plugin. It costs under an hour. It's the one fix on this list that, if missed, makes every other AI search change pointless.

Sources