robots.txt in the AI Era: Three Bot Categories, Three Different Decisions

Last updated: 1 August 2026. Bot configurations and CDN defaults change fast; this article is reviewed every two months.

If your robots.txt has a single line that “blocks AI bots,” you’re almost certainly making the wrong decision, you just don’t know it yet. AI bots are not one category but three, with separate user-agents and separate business consequences. A training crawler, a search crawler and a user-triggered agent do entirely different things, and one directive collapsing them into a shared block folds three separate questions into one answer (guptadeepak.com, July 2026).

The single most important thing to remember: if in an “I don’t want AI” reflex you block GPTBot, OAI-SearchBot, ClaudeBot and Claude-SearchBot together, you’ll succeed at blocking training but also close the only channel through which ChatGPT and Claude cite you in answers (jangwook.net, July 2026). Blocking a training crawler is legitimate content protection against model training and doesn’t reduce citability. Blocking a search crawler is voluntarily vanishing from AI results. Those are two different decisions and must be made separately.

This article breaks the three categories into concrete user-agents, shows the effect of each block, and gives a copy-ready configuration. Note: this is about how to configure access. Whether to block AI bots at all, and the arguments on both sides, we cover in a separate decision article for leadership.

Three bot categories, three different effects

The key shift of the last year or so: the major vendors split training from search into separate bots. GPTBot is not OAI-SearchBot. ClaudeBot is not Claude-SearchBot. Each has its own user-agent and can be controlled independently (DigitalApplied, June 2026). It’s precisely this split that turned the old “block all AI bots” advice into a quiet, self-inflicted loss of visibility.

Training crawlers. They fetch content to build model training datasets. Main user-agents: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), and the Google-Extended directive (a control token, not a standalone crawler; Googlebot still does the actual fetching). You may legitimately block them. Blocking does not affect AI search citations (Semola Digital, July 2026).

Search/retrieval crawlers. They fetch your pages to build the index a model cites from, with clickable sources. Main user-agents: OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity). Block OAI-SearchBot and you vanish from ChatGPT search results. That’s usually the last thing you want if you care about AI presence.

User-triggered agents. They fetch a page in real time when someone explicitly asks an assistant to read a URL. User-agents: ChatGPT-User (OpenAI), Claude-User (Anthropic), Perplexity-User. OpenAI’s docs note that because these are user-triggered, robots.txt rules may not apply. It’s usually worth allowing them, as they handle live citations during assistant browsing.

One important mistake to avoid: the older user-agents anthropic-ai and Claude-Web are deprecated. If you only block those, you’re shooting at a target that no longer exists (jangwook.net, July 2026).

Decision table: which bot, what happens when you block it

Scale of the phenomenon: who gets blocked most

Blocking AI bots is not a niche. GPTBot is the most-blocked AI crawler, appearing in 5.52% of DISALLOW rules in Q1 2026, ahead of CCBot (5.08%), ClaudeBot (4.88%), Google-Extended (4.44%) and Bytespider (4.23%) (DigitalApplied, June 2026). Some of those blocks are deliberate and justified. Some are leftovers from the 2023-2024 wave, when companies blocked OpenAI fearing training scrapes, not realising they were closing their own path to citations in AI search.

Critical note: Cloudflare defaults change on 15 September 2026

Here is the change that will make some companies block AI bots without making any decision. From 15 September 2026, new domains onboarding to Cloudflare get the training and agent categories blocked by default on pages that display ads, while the search category stays allowed (CLSkills Hub, July 2026). This means access control shifts partly from your robots.txt to the CDN’s default configuration.

The practical takeaway: you control bot access on two layers today. robots.txt tells well-behaved bots what they may fetch. Cloudflare (or another CDN) enforces the decision against bots that never ask. Those two layers must agree. Research on Cloudflare’s network found that a meaningful share of sites accidentally block major AI crawlers at the CDN level while their robots.txt says “allow” (dataimpulse.com, July 2026). If your robots.txt and CDN config disagree, the CDN wins, and you don’t see it.

There’s one more catch that should shape the whole decision: Googlebot crawls for both search and Google’s AI features. A too-broad training block via CDN controls can accidentally catch Googlebot and hurt your regular rankings (CLSkills Hub, July 2026). Never block Googlebot.

Copy-ready configuration: block training, allow search

The most common sensible policy for companies that care about citability: block training crawlers, allow search crawlers and user agents.

# Training crawlers: block
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Search crawlers and agents: allow
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Never block Googlebot
User-agent: Googlebot
Allow: /

There’s one technical trap that’s easy to hit. Some robots.txt parsers work on a first-match basis. With such a parser, an Allow: / rule placed earlier can shadow a later Disallow. A practitioner who deployed exactly this policy described having to place the Disallow lines before Allow and verify identical behaviour across both parser families (jangwook.net, July 2026). After any robots.txt change, check that it behaves the way you intend.

robots.txt’s limit: it’s a request, not a lock

This needs to be said plainly. robots.txt (standardised as RFC 9309) is a voluntary request that well-behaved crawlers honour. It has no legal force on its own and technically blocks nothing (dataimpulse.com, July 2026). Real enforcement happens at the server or CDN.

Most compliant AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) respect robots.txt at effectively one hundred percent in measurements. But there are exceptions. Cloudflare published a report in August 2025 documenting that Perplexity uses undeclared crawlers with rotating user-agents and IPs that circumvent robots.txt directives, and subsequently de-listed Perplexity from its Verified Bots programme (Semola Digital, July 2026). If you want to hard-block a bot that doesn’t listen, robots.txt isn’t enough, you need a server- or CDN-level rule (for example a 403 response or IP-range filtering).

FAQ

Which AI bots should I block?
It depends on the goal. To protect content from model training while keeping AI search presence, block training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) and allow search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Never block Googlebot.

Will blocking GPTBot remove me from ChatGPT?
Not from search. GPTBot is a training crawler. Presence in ChatGPT search results is handled by OAI-SearchBot. Blocking GPTBot protects content from training but doesn’t touch citability. Only blocking OAI-SearchBot removes you from ChatGPT results.

How do I check what’s in my robots.txt?
Visit yourdomain.com/robots.txt in your browser. Look for Disallow rules containing GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, or a blanket User-agent: * Disallow: /. If you block search bots or user agents, you’re invisible to those platforms.

Does robots.txt really block bots?
Only the ones that respect it. It’s a voluntary request with no legal force. Compliant bots (GPTBot, ClaudeBot, PerplexityBot) obey, but hard-blocking non-compliant crawlers requires a server- or CDN-level rule.

What does Cloudflare change on 15 September 2026?
New domains on Cloudflare get the training and agent categories blocked by default on ad-displaying pages, while search stays allowed. This means some companies will block bots without a conscious decision, through the CDN’s default config alone. Check that your robots.txt and CDN settings say the same thing.

Does Perplexity respect robots.txt?
Not always. Cloudflare documented that Perplexity uses undeclared crawlers that circumvent robots.txt and de-listed it from the Verified Bots programme. To hard-block Perplexity you need a server- or CDN-level rule.

A single “block AI bots” line is almost always a mistake, because it collapses three different decisions into one. You can block training with no harm to citability. You block search only when you consciously give up presence in ChatGPT, Claude and Perplexity. User agents are usually worth allowing. On top of that sits the CDN layer, which from 15 September 2026 will start making some of these decisions for you unless you configure it.

Start with one step: open your own robots.txt and check whether you’re accidentally blocking search crawlers. If you want robots.txt, CDN rules and AI visibility measurement working as one coherent system, that’s what an Insightland technical SEO audit is for. The decision itself, “block or allow,” with arguments on both sides, is the subject of a separate article for leadership.

Bot configurations and user-agents verified as of 1 August 2026 against OpenAI and Anthropic documentation, DigitalApplied, guptadeepak.com, Semola Digital, jangwook.net. This category changes fast; verify against the primary source before use.