Key takeaways
- GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot are distinct crawlers — blocking one does not block all, and allowing Googlebot does not allow AI bots
- Accidental blocks often come from SEO plugins, staging configs, or copied enterprise robots.txt templates
- Allowing AI bots does not hand over training rights in all cases — robots.txt governs crawl access; terms of use govern usage
- Audit your live robots.txt in production, not staging — CDN caches can serve outdated files
- After fixing access, expect 14–30 days before content changes flow into model retrieval behaviour
The most expensive GEO mistake is also the simplest: a Disallow rule in robots.txt that stops AI crawlers from reading your site. You can publish world-class content, deploy llms.txt, and perfect your schema — and still be invisible in ChatGPT, Claude, and Perplexity because bots were never allowed to fetch your pages.
This guide explains which bots matter, how to read your robots file, and how to fix blocks without opening the door to scrapers you still want to exclude.
Which AI Crawlers Matter in 2026
| User-agent | Operator | Why it matters |
|---|---|---|
| GPTBot | OpenAI | Training and browse/index paths for ChatGPT |
| OAI-SearchBot | OpenAI | Search-oriented crawling (newer search integrations) |
| ClaudeBot / anthropic-ai | Anthropic | Claude retrieval and indexing |
| PerplexityBot | Perplexity | Live citation and answer sourcing |
| Google-Extended | Gemini / AI Overviews corpus (separate from Googlebot) | |
| Bytespider | ByteDance | Various AI products — optional by risk appetite |
| Amazonbot | Amazon | Alexa / Amazon AI surfaces |
For most B2B and SaaS sites, GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are the core four to allow unless you have a legal reason not to.
Googlebot ≠ AI access
Allowing Googlebot does not automatically allow Google-Extended or OpenAI/Anthropic bots. Each user-agent needs explicit rules — or a careful wildcard strategy.
How to Read Your robots.txt in 60 Seconds
- Open
https://yourdomain.com/robots.txtin a browser (use production, not staging). - Look for
User-agent:blocks targeting the bots above. - If you see
Disallow: /under those agents, you are fully blocked. - If there is no mention of AI bots, you may inherit the
*wildcard rule — check whatUser-agent: *allows.
A permissive pattern for marketing sites that want AI visibility:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Your file may already end with User-agent: * rules — order and specificity matter. When unsure, use the checker below rather than guessing.
Instantly see allow/block status for 9 major AI crawlers
AI Bot Checker
Checks 9 crawlers · see your robots.txt rules instantly
Why Sites Accidentally Block AI Bots
- SEO plugins shipped “block AI training” toggles in 2024–2025. Many defaulted to on.
- Copied templates from security-conscious enterprises block all non-Google bots.
- Staging → production drift where marketing never updated robots after launch.
- CDN edge rules serving a different robots file than your CMS shows in admin.
~1 in 4
marketing sites in GEO audits had at least one major AI crawler fully disallowed — often unintentionally
GeoFundamentals sample audits, Q1–Q2 2026
Allowing Bots vs Licensing Your Content
robots.txt controls whether a crawler may fetch a URL. It does not replace your website terms, copyright policy, or platform contracts.
Practical framing for leadership teams:
- Allowing crawl = you want the option to appear in AI answers when your content is relevant
- Blocking crawl = you opt out of that discovery path for that bot
- Legal/compliance = separate conversation with counsel (regulated industries, paywalled IP, etc.)
If legal approves AI visibility, technical teams should align robots, llms.txt, and schema — not fight each other.
After You Unblock: What to Expect
| Phase | Timeline |
|---|---|
| Crawler revisits | Days 3–14 |
| New pages ingested into retrieval | Days 14–30 |
| First measurable brand citation (median) | ~38 days |
| Traffic visible in GA4 AI channels | Days 60–90 |
Unblocking is step zero. You still need answer capsules, llms.txt, and FAQ schema for competitive queries.
Once bots can crawl, give them a map of your site with llms.txt
llms.txt Generator
Auto-fills from your URL · ready to deploy in 2 minutes
AI bot access checklist
- Run AI Bot Checker on your live domain
- Fix Disallow rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended
- Re-test after CDN/cache purge if changes do not appear
- Deploy or update llms.txt at domain root
- Run GEO Score to confirm technical signals pass
- Set citation monitoring on 3–5 buyer keywords
Perplexity vs ChatGPT: Does robots.txt Differ?
The same robots.txt rules apply to both — PerplexityBot is a separate user-agent from GPTBot. Sites that unblocked OpenAI after a news cycle but never touched Perplexity remain invisible in Perplexity answers.
If your GEO strategy mentions only ChatGPT, audit all three consumer engines your buyers use. Buyer surveys vary by industry: B2B SaaS often sees ChatGPT + Perplexity; developer tools see more Claude.
Weekly checks across ChatGPT, Claude, and Perplexity — not just one engine
AI Citation Alert
ProWeekly digest · monitors ChatGPT, Claude, Perplexity · Pro plan
Frequently Asked Questions
Will allowing GPTBot increase scraper spam? robots.txt is not a security boundary. Use WAF/rate limits for abuse; use robots for crawler policy.
Can I allow Perplexity but block training bots? Policy is per user-agent. Some sites allow PerplexityBot and OAI-SearchBot while restricting others — work with legal on specifics.
Do I need to update robots when I add new sections?
Usually no — Allow: / covers new paths. Exception: intentional Disallow on /admin, /cart, etc.
How do I verify the fix worked? AI Bot Checker + GEO Score technical signal + citation monitoring over 4–6 weeks.
Does this affect GDPR? Crawl policy is one input; privacy compliance is separate (consent, data processing). Consult counsel for EU operations.