OpenAI, Anthropic, and Perplexity each run separate bots for training, search, and fetching a page because a person asked. Blocking by company name gets the result wrong. Decide by job, then write one robots.txt rule per bot.
The bots, by company and job
Each row below comes from the company’s own documentation.
| Company | Bot | Job | What blocking it does |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Surfaces sites in ChatGPT search | Site is not shown in ChatGPT search answers, though it can still appear as a navigational link |
| OpenAI | GPTBot | Crawls content that may be used for training | Signals content should not be used for training |
| OpenAI | ChatGPT-User | Fetches pages on a user’s request | robots.txt rules may not apply |
| Anthropic | Claude-SearchBot | Improves the quality of search results | Content is not indexed for search, which may reduce visibility |
| Anthropic | Claude-User | Fetches pages when a user asks Claude a question | Content is not retrieved for user queries |
| Anthropic | ClaudeBot | Collects content that may be used for training | Future material is excluded from training |
| Perplexity | PerplexityBot | Surfaces and links sites in Perplexity search | Not stated. Perplexity recommends allowing it |
| Perplexity | Perplexity-User | Fetches a page for a user’s question | Generally ignores robots.txt rules |
| Googlebot | Crawls for Google Search, including its AI features | Controls crawling for Search | |
| Google-Extended | Token for Gemini training and grounding | No effect on Google Search |
OpenAI
OpenAI’s crawler documentation says each setting is independent. You can allow OAI-SearchBot to be eligible for ChatGPT search and disallow GPTBot to keep content out of training. It also states that ChatGPT-User is not used to crawl automatically or to decide search inclusion, and that because a person triggers it, robots.txt rules may not apply. Changes to robots.txt can take about 24 hours to take effect for search.
Anthropic
Anthropic’s help center page names 3 bots. ClaudeBot collects training content. Claude-User supports people asking Claude questions, and disabling it may reduce your visibility for user-directed web search. Claude-SearchBot improves the relevance and accuracy of search responses, and disabling it prevents indexing for search, which may reduce visibility in results.
Anthropic says its bots follow robots.txt, respect anti-circumvention technology such as CAPTCHAs, and support the nonstandard Crawl-delay directive. It also says to opt out through robots.txt rather than by blocking IP addresses, because that can stop its bots from reading robots.txt in the first place.
Perplexity
Perplexity documents 2 agents. PerplexityBot is designed to surface and link websites in Perplexity search results and is not used to crawl content for AI foundation models. Perplexity-User supports user actions: when someone asks a question, it may visit a page and link to it in the answer. Perplexity says that since a user requested the fetch, this fetcher generally ignores robots.txt rules.
Its page also includes firewall guidance. If you use a web application firewall, you may need to explicitly allow both agents by matching the user agent together with the IP ranges Perplexity publishes.
Google treats AI inside Search as part of Search. Its documentation on AI features says the robots.txt directives for Googlebot are the control for how your site is crawled for Search, and that nosnippet, data-nosnippet, max-snippet, or noindex limit what is shown from your pages.
Google-Extended is separate. Google’s crawler list says it has no HTTP user agent of its own, because crawling uses existing Google user agents and the token is only a control in robots.txt. It manages whether crawled content may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google states it does not impact inclusion in Google Search and is not a ranking signal.
Microsoft
Microsoft does not publish a bot-per-job table in the pages we reviewed. What it documents is page-level controls. In Sep 2023 it said content without the nocache or noarchive tags may be included in Bing Chat answers and used to train its generative AI models. Content with nocache can be included, but only the URL, title, and snippet are shown. Content with noarchive is not included or linked in those answers. Pages with these tags still appear in Bing search results. If a page carries both tags, Microsoft says it treats the page as nocache. For paywalled pages it recommended adding nocache to noarchive, so Bing Chat users can still find the article.
In Feb 2026 Microsoft introduced an AI Performance report in Bing Webmaster Tools showing how often a site is cited in Microsoft Copilot, AI-generated summaries in Bing, and some partner integrations. It states that Bing respects content owner preferences expressed through robots.txt and other supported controls.
Checking what actually visits you
Your server logs or CDN analytics will show which of these bots reach your site and how often. Search the user agent field for the bot names in the table. A bot that never appears is either not interested in your site yet or is being stopped before it arrives.
A name in a log is easy to fake, so verify before you treat a request as real. OpenAI and Perplexity publish IP ranges for each of their agents, and Anthropic says a crawler coming from an address on its published list is from Anthropic. If you build firewall allow rules, match the user agent and the IP range together, as Perplexity’s page recommends.
Allow time after any change. OpenAI says about 24 hours for search. Google says a recrawl can take from several days to several months, depending on how often it decides a page needs refreshing. A rule that looks ignored a day later may just not have been read yet.
Choosing rules by goal
If you want to be eligible for AI answers, allow the search bots: OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot. Making sure your CDN and firewall let them through matters as much as robots.txt does.
If you want to keep your content out of model training, block the training bots: GPTBot, ClaudeBot, and use Google-Extended. This is a separate decision from being cited, and the documentation above says it does not remove you from search answers.
If you need a page to be unreachable, robots.txt is not enough. Several of these fetchers act on a person’s request and may ignore it. Put the page behind a login.
A starting point for a site that wants search visibility and no training use:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /Review it against each company’s page before you ship it. These lists change, and a bot name that is accurate today can be replaced next year. Our guide to AI crawlers and robots.txt is kept current, and the llms.txt guide covers the file people often confuse with robots.txt.
