How to manage AI crawlers in robots.txt
The Doily team
Reviewed by the Doily team, Oct 2026 Updated 4 min read
Quick answer
To appear in AI search answers, allow the search bots in robots.txt and in your firewall: OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot. Training bots such as GPTBot, ClaudeBot, and Google-Extended are separate choices. Each vendor documents its own user agents, so set rules bot by bot.
Why it matters
Most AI engines use more than 1 bot, and each bot has a different job: building a search index, fetching a page when a person asks a question, or collecting content for model training. A single blanket rule either hides you from answers or hands over more than you meant to.
Blocks also arrive by accident. A security plugin, CDN setting, or old robots.txt can shut out a search bot without anyone deciding to. Checking takes 20 minutes and removes a cause you cannot see from rankings.
Key takeaways
- Separate 3 purposes: search, user-triggered fetching, and training. Decide each one on its own.
- Blocking a training bot does not remove you from the same vendor's search results. OpenAI states its bot settings are independent.
- robots.txt is a request that crawlers choose to follow. It does not secure private content.
- User-triggered fetchers may not follow robots.txt, according to OpenAI and Perplexity.
Step by step
-
Read what is live today
Open yourdomain.com/robots.txt and read every rule. Then check your CDN, firewall, hosting panel, and security plugins for bot rules you did not write. Note any blanket block of unknown bots, since it can catch search bots too.
-
Decide a policy for each purpose
For search, allow the bots that build the indexes behind AI answers if you want to be found there. For user-triggered fetches, allowing them lets an engine open your page when a person asks about it. For training, choose based on your business. Google says its training token, Google-Extended, has no effect on Search inclusion or ranking.
-
Write the rules per user agent
Add a separate group for each bot, using the exact token from the vendor's documentation. Crawlers that follow the standard use the group that names them, so a named group takes priority over
User-agent: *. The example below shows the format. -
Allow the bots through your firewall
If you use a web application firewall, allow search bots by user agent and by the published IP ranges. OpenAI and Perplexity publish JSON files of their ranges. Matching both the name and the address avoids letting impostors in.
-
Wait, then confirm in your logs
OpenAI says search systems can take about 24 hours to pick up a robots.txt change. After a few days, scan your server logs for each bot's user agent to confirm it is fetching pages and getting 200 responses.
-
Know what robots.txt cannot do
Google says robots.txt is not a way to keep a page out of search results, and that a blocked URL can still appear without a description. Use
noindexor a login for that. Anthropic warns that blocking IP addresses can stop its bots from reading robots.txt at all. -
Review on a schedule
Vendors add and rename bots. Re-read the vendor pages every quarter and update the file. Keep a dated note of what you allowed and why.
Bots and a starting file
| Vendor | User agent | What it does | robots.txt control |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Surfaces sites in ChatGPT search | Yes |
| OpenAI | GPTBot | Crawls content that may train models | Yes |
| OpenAI | ChatGPT-User | Visits pages for user actions | OpenAI says rules may not apply |
| Anthropic | Claude-SearchBot | Improves search result quality | Yes |
| Anthropic | ClaudeBot | Collects content for model training | Yes |
| Anthropic | Claude-User | Fetches pages when a user asks | Yes |
| Perplexity | PerplexityBot | Surfaces and links sites in Perplexity search | Yes |
| Perplexity | Perplexity-User | Visits pages for user questions | Perplexity says it generally ignores rules |
| Googlebot | Crawls for Search, including AI Overviews and AI Mode | Yes | |
| Google-Extended | Token for Gemini training and grounding | Token only | |
| Apple | Applebot | Powers search in Spotlight, Siri, and Safari | Yes |
| Apple | Applebot-Extended | Opt-out for training Apple models | Token only |
A starting file that allows search and user-triggered access and blocks training. Change the second half if you decide differently.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /Googlebot is not listed because it already has access under your normal rules. Blocking Google-Extended or Applebot-Extended is a training choice and does not remove you from their search products.
How Doily helps
Doily's site audit crawls your site and ranks fixes by priority, and AI visibility tracking shows whether the engines you allowed are citing your pages.
Questions
Will blocking GPTBot remove me from ChatGPT search?
No. OpenAI says its settings are independent. OAI-SearchBot controls search results, and GPTBot controls training use. Sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links.
Does blocking Google-Extended affect Google Search?
No. Google says Google-Extended does not affect inclusion in Search and is not a ranking signal. It covers training for Gemini models and grounding in Gemini Apps and Vertex AI.
What about Copilot, Grok, and DeepSeek?
This guide lists only the bots verified from vendor documentation on Oct 11, 2026. Check Microsoft, xAI, and DeepSeek for their current published crawler details before writing rules for them.
Sources
Primary sources, checked Oct 11, 2026.
