Should Your Tech Company Block AI Crawlers? GPTBot, ClaudeBot and Robots.txt

Should you block AI crawlers? Block the training bots if you object to model training, and keep the search bots that can surface your pages in ChatGPT, Claude and Perplexity answers. Each bot gets its own robots.txt rule.

Open your robots.txt file. Go on, it’s one URL. If somebody pasted a block of rules into it last year, there’s a decent chance nobody has read it since. Quiet file. Big consequences.

Or it goes the other way. Leadership reads a headline about AI companies scraping websites, and by lunch a developer has added a blanket Disallow for anything with “AI” in its name. Fast. Decisive. A little too broad.

It makes sense why. “Block the scrapers” sounds like protection. But the bots behind ChatGPT, Claude and Perplexity don’t all do the same job, and shutting out the wrong one can quietly remove your company from the answers a buyer reads before they ever call you. Not ideal.

Block AI Crawlers by Job, Not by Name

An AI crawler is an automated program that fetches web pages for an AI company. The operators describe three jobs. Training crawlers collect content that may be used to train models, search crawlers surface pages in an AI product’s search results, and user-requested fetchers visit a page because someone asked a question.

OpenAI’s crawler documentation says each of its settings is independent of the others, so a webmaster can allow one bot in order to appear in search results while disallowing another that gathers content that may be used for training. Anthropic and Perplexity also run separate bots for separate jobs.

Who Runs What

Read across a row and you’ll see the pattern. One company, three jobs, three separate switches. Bot names change. Check each operator’s page before you edit the file.

What Blocking a Training Bot Does

GPTBot is OpenAI’s training crawler, and its page says disallowing it indicates a site’s content should not be used in training generative AI foundation models, which is a separate switch from the one that controls search.

Anthropic words it the same way for ClaudeBot, in its help center article on blocking its crawlers, saying that restricting it signals that a site’s future materials should be excluded from its training datasets. Note the word “future”.

Google-Extended is the odd one out. It has no user agent string of its own, so it’s not a crawler you’ll find in your logs. It’s a token in robots.txt that manages whether content Google crawls may be used to train future Gemini models and to ground answers in Gemini Apps and in Grounding with Google Search on Vertex AI. Google’s list of common crawlers says it doesn’t affect a site’s inclusion in Google Search and isn’t a ranking signal.

So here’s what a training block is, going by the operators’ own words. A stated preference about how future models are built. Not an exit from ChatGPT search, Claude search or Google. Worth knowing.

The syntax is also converging. Anthropic says that as of October 30, 2026 it recognizes a “no AI training” signal based on a draft from the IETF’s aipref working group. Microsoft’s October 2026 announcement on training and grounding preferences describes a robots.txt group, User-Agent: __usage_ai-training followed by Disallow: /, and says Microsoft will honor it from February 1, 2027. Google says it intends to support the same signal in early 2027. It’s still a draft, and Anthropic warns that how it reads these settings may change. Plan for that.

What Blocking a Search Crawler Costs You

This is where blanket rules hurt. Keep these. The search bots are the ones that let an AI product find your page and link to it.

• OpenAI says sites opted out of OAI-SearchBot won’t be shown in ChatGPT search answers, though they can still appear as navigational links. It recommends allowing the bot.
• Anthropic says disabling Claude-SearchBot may reduce your visibility and accuracy in search results. Disabling Claude-User may reduce visibility for user-directed search.
• Perplexity describes PerplexityBot as the bot built to surface and link sites in its results, and recommends allowing it.
• Microsoft says choosing to disallow grounding will mean reduced brand visibility in AI experiences, and that no choice comes with a promise of inclusion, traffic or payment.

Google is the one to get right. Its guide to AI features and your website says there are no extra requirements to appear in AI Overviews or AI Mode, and that a page must be indexed and eligible to show with a snippet. It lists allowing crawling in robots.txt, and by any CDN or hosting infrastructure, among the fundamentals.

What happens once the bot can reach you is a separate job. We cover it in our post on how to rank in ChatGPT answers.

Robots.txt Is a Request, Not a Lock

The standard itself says so. RFC 9309, the Robots Exclusion Protocol, states that its rules are not a form of access authorization, that the protocol is not a substitute for valid content security, and that listing a path in the file makes the path discoverable. Google’s introduction to robots.txt agrees: the file can’t enforce crawler behavior, and if you want information kept secure, use password protection.

Then there are the fetchers. OpenAI says ChatGPT-User visits are triggered by a person, so robots.txt rules may not apply, and Perplexity says in its crawler documentation that Perplexity-User generally ignores robots.txt for the same reason, because a user requested the fetch.

A Disallow line is a polite sign on an unlocked door. Some visitors read it. Some don’t.

Anthropic adds a wrinkle. It says its bots honor robots.txt, and that blocking its IP addresses may not give a lasting opt-out, because it stops the crawler from reading the file at all. That’s a tidy example of a firewall rule undoing a robots.txt rule.

So client portals, gated reports, staging sites and anything under an NDA need a login. Not a line in a text file.

Should You Block AI Crawlers on a Tech Company Site? A Decision Rule

Here’s the rule we’d start with for a marketing site whose job is to get found by buyers. Treat it as a starting point, not a verdict.

• Search crawlers. Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot. These are the bots that let an AI product find your pages and link to them when someone asks.
• User-requested fetchers. Leave them alone. Claude-User you can control, and the other two may ignore the file anyway.
• Training crawlers. Your call, and it’s more a contracts and values question than a marketing one. Block if your published content is the product (paid research, licensed data), if a client agreement restricts reuse, or if leadership objects on principle. No operator page we read ties the training setting to search visibility, and OpenAI and Google both say the settings are independent.
• Private material. A login, always.

If you decide to opt out of training and nothing else, the file might look like this.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

The Allow lines are optional, because allowing is the default. We’d keep them anyway, so the next person who opens the file can see the intent.

Check It Before You Trust It

Writing the rule is the easy half. Then verify. Five checks tell you whether the file does what you meant.

1. Read the live file at yourdomain.com/robots.txt, top to bottom, and look for a leftover blanket Disallow from a staging site or an old rule nobody owns.
2. Check every subdomain. Anthropic says to add the rule to the robots.txt of each subdomain you want to opt out, so a help center or docs site needs its own.
3. Look one layer up. A CDN, firewall or bot-protection setting can block the same bots your file allows. OpenAI recommends allowing its published IP ranges for its search bot, and Perplexity documents firewall settings for its bots.
4. Wait a day. OpenAI and Perplexity both say a change can take up to about 24 hours to reach their systems, and RFC 9309 says crawlers shouldn’t use a cached copy longer than that unless the file is unreachable.
5. Check your server logs a few days later for the bot names. Google warns that a user agent string can be spoofed, so match what you see against the operator’s published IP list.

Reachable is step one. Understood is step two, and Google says structured data should match the visible text on the page, which is the kind of work our schema markup services cover. An llms.txt file is a separate file with a separate job, and our guide to building an llms.txt file for your MSP website covers it.

Where Tech Companies Trip on This

• A blanket Disallow for every user agent that sounds like AI. That includes the search bots.
• A rule list copied from a blog post written last spring. OpenAI alone lists four bots now, including OAI-AdsBot, which only visits pages submitted as ads and isn’t used to train models.
• Rules on the main domain and nothing on the help center.
• A firewall setting the hosting team switched on, quietly blocking the bots marketing just allowed. Nobody told anybody.
• No owner. Somebody has to be on the hook for rereading these pages a few times a year.

Blocking by name isn’t a strategy. It’s a reflex. And a rule nobody owns is a rule nobody checks.

So the better question isn’t whether to block AI crawlers. It’s which ones, for what, and who checks. We take the wider question of showing up in AI answers apart in our post on getting a tech company into AI answers. If you’d like a second set of eyes on your robots.txt before the next edit, book a growth strategy call.