robots.txt tester.
Paste your robots.txt, enter a crawler name such as Googlebot or GPTBot, and list the paths to test. The tool picks the group for that crawler, finds the longest matching rule, and tells you whether each path is allowed or blocked, with the line that decided it.
Paste the file's contents.
A user agent name such as Googlebot or GPTBot, or a full user agent string.
One per line, up to 100.
Matching follows RFC 9309: the group for the crawler's name is used, else the group for *. Within the group the longest matching rule wins, and Allow wins a tie. This tests the text you paste. It does not fetch your site, and a crawler may apply extra rules of its own.
This tool runs in your browser. Nothing you enter is sent anywhere or stored.
How to use it
- Paste the file Copy the contents of your robots.txt. The tool reads the text you paste and does not fetch your site.
- Enter a crawler Use a name such as Googlebot, Bingbot, or GPTBot, or paste a full user agent string.
- List paths to test Add one path or full URL per line, up to 100.
- Read the verdicts Each row shows allowed or blocked and the rule and line that decided it.
Why it matters
A robots.txt file tells crawlers which paths they may fetch. Rules sit in groups, and a crawler follows the group that names it, or the group for the star if none does. Inside the group, the longest matching rule wins, and Allow wins a tie. That is the standard set out in RFC 9309, and Google documents the same behavior.
Small mistakes cost a lot. One stray slash can block a whole site, and a rule meant for 1 crawler may not apply to another because groups are not merged across different names. Testing paths against the file before you deploy it catches these cases early.
Remember what robots.txt does not do. It controls crawling, not indexing, so a blocked page can still appear in results if other sites link to it. To keep a page out of results, allow it to be crawled and use a noindex tag. This is also the file where you set rules for AI crawlers.
Sources
Questions
Does this fetch my live robots.txt?
No. It tests the text you paste, so you can check a change before it goes live.
How are wildcards handled?
A star matches any run of characters and a dollar sign matches the end of the path, as the standard describes.
Will a blocked page stay out of Google?
Not always. Blocking stops crawling, but a page can still be listed if other pages link to it. Use noindex to keep it out of results.
Can I test AI crawlers?
Yes. Enter a name such as GPTBot, ClaudeBot, or PerplexityBot, as long as your file has rules for it or for the star.
Keep going
Read the Doily guides for step-by-step playbooks. To track results over time, see Doily's SEO tools.
