Which sites block AI crawlers? A 28-domain access benchmark

AI visibility starts with a simple eligibility question: can the relevant retrieval system access the page? Blocking a platform's search crawler can remove the page from that retrieval path, but allowing it never guarantees a citation. For ChatGPT Search the relevant control is OAI-SearchBot, not GPTBot. Google AI Overviews and AI Mode use Google's core search systems, so Googlebot access matters; Google-Extended does not control inclusion in Google Search.

What the benchmark tests

Date of observation: 2026-08-21. Sample: 28 domains across six categories — AI companies, major SaaS, media publishers, e-commerce, social platforms, and reference sites. The full domain list appears below so anyone can rerun the test.

For each domain, /robots.txt and /llms.txt were fetched over HTTPS. robots.txt was parsed into per-user-agent groups rather than searched as a string. A crawler counts as blocked when its named group, or the wildcard (*) group when no named group exists, disallows the root path (/) and no matching root allow rule overrides it. This is the same rule used by AEO Mantis's site audit.

The five scored crawlers are tied to search or citation retrieval:

• OAI-SearchBot — enables discovery and inclusion in ChatGPT Search
• PerplexityBot — builds Perplexity's search index
• Claude-SearchBot — accesses web content for Anthropic's search experiences
• Googlebot — builds the Google Search index used by AI Overviews and AI Mode
• Bingbot — builds Bing's search index

Results: which domains block which crawlers

The table shows every domain in the sample. ✓ means the crawler is allowed; ✗ means robots.txt blocks it from the entire site. The llms.txt column shows whether the domain publishes one.

Which crawlers get blocked most

Googlebot and Bingbot are almost universally allowed — only Reddit blocks either one. Dedicated answer-engine crawlers see higher block rates:

Five domains — Amazon, The New York Times, X, Figma, and Reddit — block all three dedicated answer-engine crawlers in the sample. Wired and The Guardian allow OAI-SearchBot but block PerplexityBot and Claude-SearchBot. These splits show why one generic “AI crawler” rule is not enough: access must be evaluated against the exact retrieval path a platform uses.

GPTBot and Google-Extended can still matter to a publisher's content-use policy, but they answer different questions. Blocking GPTBot opts content out of potential OpenAI model training while leaving OAI-SearchBot independently configurable. Google-Extended does not determine Google Search, AI Overview, or AI Mode eligibility, so neither token belongs in a citation-readiness score.

llms.txt adoption remains low

The llms.txt convention is an emerging standard where sites publish a plain-text file pointing AI systems at their most citable pages. In this sample, only 11 of 28 domains serve one — roughly 39%. The adopters skew toward developer-tooling companies (Stripe, Shopify, Vercel, Cloudflare, Notion, GitHub, Linear, Supabase) rather than media publishers, none of whom publish one.

This is not surprising for an early convention. Its absence is not a citation blocker: major search and answer platforms do not require llms.txt for eligibility. It can still be a useful optional index for tools that choose to consume it, so the checker notes the file when present without scoring its absence.

What this means for AI visibility

The practical takeaway is narrower than “allow every AI bot.” Match each desired visibility surface to its actual retrieval crawler. If you want ChatGPT Search eligibility, check OAI-SearchBot; for Google AI search features, protect normal Googlebot crawlability. Keep model-training controls separate. Access only creates eligibility—relevance, authority, quality, indexing, and answer selection still determine whether a page is cited.

Three patterns emerged in the data that are worth checking against your own site:

• Unintentional wildcard blocks. A robots.txt that disallows / for User-agent: * blocks every crawler that does not have its own named group. If you added this rule for a specific crawler and forgot it also applies to the wildcard, you may have blocked more agents than intended.
• Inconsistent per-crawler policies. Several domains in this sample allow some AI crawlers but block others. If your team allowed one named crawler but did not update robots.txt as newer crawlers appeared, they may be falling through to a restrictive wildcard rule.
• Optional discovery files treated as requirements. llms.txt may help tools that consume it, but its absence should not lower a citation-readiness score or trigger a blanket remediation.

How to check your own site

The free AI visibility checker runs the same parser on any URL you enter. It tests all five retrieval crawlers, notes an optional llms.txt when present, and reviews extractability and Organization markup. No sign-up required — one URL in, a prioritized fix list out.

For a full picture of whether AI platforms actually mention and cite your brand — not just whether they can crawl your site — AEO Mantis monitoring runs real prompts across six AI platforms on a schedule and tracks mentions, citations, competitors, and sources over time.

Limitations

This benchmark tests one binary dimension: does robots.txt contain a site-wide block for the named crawler? It does not verify crawler IPs, CDN or WAF behavior, actual visits, indexing, rendered extraction, or whether content earns citations. A domain that allows all five may still be absent from answers. The full picture requires monitoring actual AI answers, not just checking robots.txt.

The sample is 28 domains chosen for category diversity, not statistical representativeness. The rules were fetched once on 2026-08-21 and may change. midjourney.com returned HTTP 403 and stackoverflow.com returned HTTP 418 for robots.txt; the table shows no detected block for them because no rules were available to parse, which is not the same as confirming that their infrastructure allows these crawlers.