Can AI Crawlers Actually Reach Your Site?

There is a specific kind of wasted quarter I keep running into. A company decides AI search matters. They rewrite their content into clean question-and-answer passages, add FAQ schema, tighten their headings. Six months later nothing has changed, and nobody can say why.

Then I check robots.txt and find GPTBot disallowed — added two years earlier by someone who read an article about protecting content from AI training, left the company, and never mentioned it.

Everything downstream of that line was wasted effort. A model cannot quote a page it was never allowed to fetch.

Access is the first gate, and it is binary#

Optimising for AI citation has a dependency chain, and the order is not negotiable:

1. The crawler can fetch the page
2. The page is in a retrievable index
3. The content is structured so a passage can be extracted cleanly
4. The passage is credible enough to be selected

Steps two through four are craft. Step one is a switch. If it is off, the rest of the work produces exactly zero — not diminished returns, zero. This is why I check it before quoting on any GEO and AEO engagement, and why it is the first item in the free audit.

The crawlers that matter, and what each one controls#

Bot names get confused constantly, usually because they sound similar and do completely different jobs. Here is what each one actually governs.

Two rows in that table cause most of the confusion.

Google-Extended does not control AI Overviews. This is the single most common mistake I see. AI Overviews is generated from the standard Google Search index — the one regular Googlebot builds. Google-Extended is a separate control that governs whether your content trains and grounds Gemini. Blocking it does not remove you from AI Overviews. If you want out of AI Overviews specifically, the mechanism is nosnippet or max-snippet, and those cost you regular search snippets too.

Applebot-Extended is not Applebot. The plain Applebot powers Siri and Spotlight search. The -Extended variant is the Apple Intelligence training control. Blocking the wrong one removes you from Apple's search surfaces for no benefit.

Ten minutes to check it properly#

Read your robots.txt like a crawler does#

Open yourdomain.com/robots.txt and look for two things: an explicit Disallow under any AI user-agent, and a catch-all User-agent: * block that disallows more than you remember.

The order matters. Most crawlers match the most specific applicable group and ignore the rest. So this permits GPTBot everywhere despite the wildcard rule:

User-agent: *
Disallow: /

User-agent: GPTBot
Allow: /

And this blocks it, because the specific group wins:

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

If you cannot tell at a glance which groups apply, that is itself the finding. A robots.txt nobody can read confidently is one incident away from blocking something important.

Then test the actual response#

robots.txt is a request, not an enforcement layer. Your CDN, WAF, or bot-management rules sit in front of it and can refuse the connection before robots is ever consulted. This is the failure mode that silently kills more AI visibility than robots ever has.

Request your own page while identifying as each crawler:

curl -sS -A "GPTBot/1.0" -o /dev/null -w "%{http_code}\n" https://yourdomain.com/
curl -sS -A "ClaudeBot/1.0" -o /dev/null -w "%{http_code}\n" https://yourdomain.com/
curl -sS -A "PerplexityBot/1.0" -o /dev/null -w "%{http_code}\n" https://yourdomain.com/
curl -sS -A "OAI-SearchBot/1.0" -o /dev/null -w "%{http_code}\n" https://yourdomain.com/

You want 200 from every one. A 403 means something at the edge is refusing them. A 503 usually means a bot-challenge page. Both are invisible in robots.txt and invisible in analytics, which is why they survive for years.

Check a service page and a blog post as well as the homepage. Rules are often path-scoped, and it is common to find the marketing site open while everything under /blog/ sits behind a challenge.

Confirm the content is in the HTML#

The last check is what actually comes back. Some AI crawlers execute JavaScript unreliably or not at all. If your key content renders client-side, a crawler can get a clean 200 and still receive an empty shell.

curl -sS -A "GPTBot/1.0" https://yourdomain.com/your-best-page | grep -c "a distinctive phrase from that page"

A 0 means the page is technically reachable and practically empty. That is a rendering problem, and it belongs to structure and infrastructure work rather than to content.

Where silent blocks actually come from#

In every case I have diagnosed, the block came from one of four places, and none of them were a deliberate content-licensing decision:

• Cloudflare bot settings. Bot Fight Mode and the newer AI-crawler toggles are a single click and are frequently enabled during a scraping incident, then forgotten.
• WAF rules added during an attack. Someone blocks a range of user agents at 2am to stop a problem. The rule stays for three years.
• A hosting default. Some managed WordPress hosts ship aggressive bot rules that include AI crawlers, enabled out of the box.
• A robots.txt line inherited from a template. Copied from a boilerplate or a "protect your content" blog post, never revisited.

The common thread is that nobody chose this as strategy. It is a side effect that survives because nothing surfaces it. There is no alert, no Search Console warning, no report. You find it by looking.

Then decide, deliberately#

Once you know your actual state, make it a decision rather than an accident.

Allow, if you sell anything researched before purchase. Being named as the source in an AI answer is the modern equivalent of ranking — it is a recommendation, delivered with more authority than a blue link.

Block, if your content is the product. Paid research, licensed data, subscription journalism. Then blocking is coherent strategy, not paranoia.

Do not do what most sites do, which is block the training crawlers, leave the search crawlers open, and assume that gets the visibility without the training. It partly works — OAI-SearchBot and Claude-SearchBot are separate from GPTBot and ClaudeBot — but the split is finer than most teams realise, and getting it wrong in the other direction is easy. If you are going to draw that line, draw it on purpose and write down why.

One more thing worth knowing: this site allows every crawler in the table above. You can read the robots rules yourself. It would be difficult to sell AI visibility work from a site that blocks the systems doing the citing.

What to do this week#

1. Open robots.txt and identify every group that applies to an AI crawler
2. curl four crawlers against three different page types and confirm 200
3. Check the returned HTML actually contains your content
4. Check your CDN's bot-management panel — this is where the surprises are
5. Write down the decision, so the next person does not have to guess

If any of those come back wrong, that is not a content problem and no amount of rewriting will fix it. Send me the URL and I will tell you which layer is refusing the request.