AI visibility
AI visibility Nine AI engines, tracked Prompt research Find the questions worth tracking Competitor analysis Who wins on Google and in AI answersSEO
SEO tools Rank tracking, backlinks, and more Keyword research Volume, difficulty, and intent Site audits A health score and ranked fixes Local SEO Map pack grids and reviewsWorkflow
Closed-loop marketing From finding to task to report Ask Doily Answers and tasks from your data Reporting Branded reports that send themselves
The Doily team Team AI engines can only cite what they can fetch. If a firewall or a robots.txt rule is turning their crawlers away, you could be missing for a reason that has nothing to do with your content. How do you check?
The Doily team Team Start with robots.txt, then check the layers in front of your site, because most blocks hide there.
1. Read robots.txt. Open yoursite.com/robots.txt and look for rules that name AI crawlers, or a blanket Disallow: / under User-agent: *. Each AI company documents its own crawler names, and they are separate bots for separate jobs: some collect training data, some build a search index, and some fetch a page live when a person asks a question. Blocking one does not block the others. Look the names up in the vendor's documentation instead of copying a list from a blog.
2. Test the real response. A robots.txt file that allows a bot means nothing if the server refuses it. Request a page with the full user agent string the vendor publishes and read the status code:
curl -I -A "<full user agent string from the vendor docs>" https://yoursite.com/A 200 is good. A 403, a 429, or a challenge page means something is in the way. Some firewalls also check the visitor's IP range against the vendor's published list, so a plain curl can pass while the real bot fails.
3. Check your CDN and firewall. Bot protection at hosting and CDN providers often challenges unfamiliar automated traffic by default. Open the firewall event log and search for the user agents you care about.
4. Check rendering. Many crawlers do not run JavaScript. View the page source, not the browser inspector, and confirm your text is in the HTML.
5. Read your server logs. They show which bots actually visit and what each one gets back.
If everything is open and you still do not appear, the cause is almost always content or reputation. Access is the entry requirement, not the whole game.
The Doily team Team Which bots to allow depends on your goal.
For search visibility, allow the search bots: OAI-SearchBot for ChatGPT search, Claude-SearchBot, PerplexityBot, and Googlebot. To keep your content out of model training, block GPTBot, ClaudeBot, and the Google-Extended token. These are separate decisions. OpenAI says its settings are independent, and Google says Google-Extended does not affect inclusion in Google Search.
OpenAI also says robots.txt changes can take about 24 hours to be reflected for search, so do not judge a change after an hour.
The Doily team Team ChatGPT-User and Perplexity-User are different. They fetch a page because a person asked a question. Both companies say robots.txt rules may not apply to them, since a user triggered the request.
If a page must stay private, put it behind a login. A robots.txt rule is a request, not a lock.
The Doily team Team If you write firewall allow rules, match the user agent together with the IP ranges the vendor publishes. A user agent string can be copied by anyone, and Perplexity recommends pairing the two.
Anthropic advises against blocking by IP address as a way to opt out. It says that can stop its bots from reading robots.txt at all, so use robots.txt rules for that.
Your pages sit on page one for the searches that matter. Then you ask ChatGPT for recommendations in your category and it names other businesses, never yours. It looks like a contradiction. What explains it, and what…
llms.txt is a plain text file at the root of a site that lists key pages for AI systems. Plenty of advice says to add one. Does it change whether AI engines mention or cite you?
You run one prompt on Monday and your business is named first. On Wednesday it is missing. Nothing on the site changed. Which answer should you believe?
