AI crawler robots.txt generator.
Choose which AI crawlers may fetch your site and get a ready-to-upload robots.txt. The tool covers 13 crawlers from OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, and Microsoft, grouped by what they do: training, search, or user-triggered fetches. Blocking one group does not block the others.
Training and model improvement
Collects pages that can be used to train or improve AI models. Blocking these keeps your content out of future training, and does not remove you from search.
Crawls content that OpenAI may use to train its models.
Collects public web content that may contribute to training Claude.
A robots.txt token, not a separate crawler. Controls whether content Googlebot crawled can be used for Gemini. Does not affect Google Search.
A robots.txt token, not a separate crawler. Controls whether content Applebot crawled can be used to train Apple models.
Builds the open Common Crawl archive that many AI models are trained on.
Search and answer indexing
Builds the index an AI search product or search engine draws from when it answers a question and links to sources.
Finds and links pages in ChatGPT search results.
Indexes pages to improve the quality of Claude search results.
Surfaces and links sites in Perplexity search results. Not used for model training.
Google Search crawler, which also feeds AI Overviews and AI Mode.
Blocking Googlebot removes your pages from Google Search.
Bing crawler. Bing results are also used by other products.
Blocking Bingbot removes your pages from Bing search results.
User-triggered fetches
Fetches a page because a person asked an assistant about it. Some vendors say robots.txt rules may not apply to these, because the request comes from a person.
Visits a page when a person asks ChatGPT or a custom GPT to.
Visits a page when a person asks Claude about it.
Visits a page when a person asks Perplexity to.
Optional. One path per line, starting with a slash.
Optional. Use the full address.
Upload the file to the root of your domain, then open yoursite.com/robots.txt to confirm it loads. The crawler list follows each vendor's own documentation, linked under Sources below.
This tool runs in your browser. Nothing you enter is sent anywhere or stored.
How to use it
- Start from a preset Allow everything, block training crawlers only, or block every AI crawler.
- Adjust each crawler Switch individual crawlers between allowed and blocked. Warnings appear before you block a search engine.
- Add private paths and your sitemap List folders every crawler should skip, and add the full sitemap address.
- Upload the file Save it as robots.txt at the root of your domain, then fetch it in a browser to confirm it loads.
Why it matters
AI companies run more than one crawler, and each does a different job. A training crawler collects pages to improve models. A search crawler builds the index behind an AI answer that links to you. A user-triggered fetcher loads a page because a person asked an assistant about it. Treating them as one thing leads to blocks that cost you visibility you wanted.
A common choice is to block training crawlers and allow search crawlers, so your content stays out of model training but can still be cited in answers. Other sites block nothing, because being cited is the goal. Neither choice is right for every site. It is a trade you make on purpose, and if you block a search crawler you should expect to lose citations from that product.
robots.txt is a request, not a lock. Well-behaved crawlers follow it, and some vendors say user-triggered fetchers may not. Content that must stay private belongs behind a login. A crawler follows the most specific group that names it, so this tool repeats your private paths inside each crawler group. Check the vendor pages below a couple of times a year, because crawlers are added and renamed.
Sources
Questions
Where does robots.txt go?
At the root of the domain, for example https://example.com/robots.txt. Each subdomain needs its own file.
Does blocking GPTBot remove me from ChatGPT answers?
OpenAI lists GPTBot and OAI-SearchBot as separate crawlers with separate jobs. GPTBot is for training and OAI-SearchBot is for ChatGPT search results, so you can block one and allow the other.
Will blocking Google-Extended hurt my Google rankings?
Google documents Google-Extended as a separate token for Gemini use, and it does not control whether you appear in Google Search. Blocking Googlebot does, which is why the tool warns you.
How can I tell which crawlers visit my site?
Look for the crawler names in your server logs. User-agent strings can be faked, so check them against the IP ranges that vendors publish.
Keep going
Read the Doily guides for step-by-step playbooks. To track results over time, see Doily's AI visibility.
