OpenAI’s crawler, OAI-SearchBot, is what surfaces websites in ChatGPT’s search features, and a separate fetcher, ChatGPT-User, visits pages when a person asks a question. OpenAI’s developer documentation says search-backed responses include inline citations by default. OpenAI publishes the crawler details, which are the part you can act on, and does not publish how it ranks pages.
The three user agents
OpenAI’s crawler documentation lists the bots it runs. Three matter for visibility.
OAI-SearchBot is for search. OpenAI says it surfaces websites in the search results of ChatGPT’s search features. Sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI recommends allowing it in robots.txt and allowing requests from its published IP ranges.
GPTBot crawls content that may be used to train OpenAI’s generative AI foundation models. Disallowing it signals that your content should not be used for training.
ChatGPT-User handles certain user actions in ChatGPT and Custom GPTs. When someone asks a question, ChatGPT may visit a page with this agent. OpenAI says it is not used to crawl the web automatically, and because a person starts the request, robots.txt rules may not apply. It is not used to decide whether content can appear in search.
OpenAI also lists OAI-AdsBot, which checks the landing pages of ads submitted to ChatGPT. It only visits pages submitted as ads.
Search and training are separate switches
The documentation is explicit that each setting is independent. A site can allow OAI-SearchBot so it shows up in ChatGPT search while disallowing GPTBot so its content is not used for model training. If both are allowed, OpenAI says it may use the results of one crawl for both purposes to avoid crawling twice.
A robots.txt that does that looks like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /OpenAI notes that changes to robots.txt can take about 24 hours to be reflected for search. Do not judge a change after an hour.
The two mistakes we would look for first are a broad rule that blocks every unrecognized bot, which catches OAI-SearchBot along with the ones you meant to stop, and a firewall that challenges or blocks the crawler before it reaches robots.txt at all. In both cases the site owner believes search is allowed and the crawler never gets a clean response.
What happens when someone asks a question
The mechanics OpenAI documents publicly are for developers. Its documentation for the web search tool describes how the tool works for API users. It separates 3 approaches. A non-reasoning search sends the question to the search tool and passes along the top results. Agentic search lets a reasoning model manage the process: it can run a search, open a page, and look for text within a page, then decide whether to keep going. Deep research is a longer version that can consult hundreds of sources over several minutes.
Two details from that page are useful even if you only care about the ChatGPT app, with the caveat that the app and the API are different products and may not behave identically.
First, location is an input. Developers can pass an approximate country, city, region, and timezone to refine results by geography. The system can take location into account, so local questions are a case to test.
Second, developers can limit the search to a list of allowed domains or exclude a list of blocked ones, up to 100 each. The model is not always searching the whole web.
Citations are not the same as sources
By default, a response that used web search includes inline citations for the URLs it relied on, and OpenAI tells developers those citations must be clearly visible and clickable in their interface. Each citation carries a URL, a title, and the position in the text where it applies.
The documentation also describes a separate list of sources: every URL the model consulted while forming its answer. It says the number of sources is often greater than the number of citations. So a page can be read, shape the answer, and never appear as a citation. From the outside, you only see the citations.
Repeat the question, get a different list
Even with identical settings, the lists differ from run to run. SparkToro and Gumshoe.ai had volunteers run 12 prompts through ChatGPT, Claude, and Google’s AI features a combined 2,961 times. They reported a less than 1 in 100 chance that ChatGPT or Google’s AI would return the same list of brands in any two responses to the same prompt. The authors disclose that Gumshoe.ai sells AI tracking tools and call for larger follow-up studies. One run tells you very little. Many runs of the same prompt show how often you appear.
Recognizing the requests in your logs
OpenAI gives example user agent strings so you can tell the bots apart. OAI-SearchBot identifies itself as OAI-SearchBot/1.4 inside a Chrome-style string, and OpenAI notes the version number may change. When it fetches robots.txt it may add a robots.txt marker to the string so you can spot those requests even if your logs do not record paths. GPTBot shows up as GPTBot/1.4 and ChatGPT-User as ChatGPT-User/1.0.
A user agent string can be copied by anyone. For anything that matters, such as allowing traffic through a firewall, match the request against the IP ranges OpenAI publishes for each bot instead of trusting the name alone.
What to check this week
- Open your robots.txt and look for rules naming OAI-SearchBot, GPTBot, or a blanket
Disallow: /for unknown bots. Remove anything blocking OAI-SearchBot unless you intend it. - Check your CDN or firewall. Bot protection rules often block unfamiliar crawlers before robots.txt is ever read. OpenAI publishes IP ranges for OAI-SearchBot at openai.com/searchbot.json, and for the other agents at similar addresses listed in its documentation.
- Search your server logs for the three user agent names. Requests from ChatGPT-User are a sign that someone asked ChatGPT about something your pages answer.
- Run the 5 or 10 questions your customers ask and read the citations. Note which of your pages appear and which competitors do. Write down the date and where you ran each one, so you can compare later runs fairly.
Our guide to AI crawlers and robots.txt goes through the file line by line.
What OpenAI does not say
The crawler page does not say how pages are ranked, how many sources go into an answer, or how long a page stays in the search index. Nothing in it promises that allowing OAI-SearchBot gets you cited. It only says that blocking it removes you. Allowing the crawler is the entry condition, not the strategy. What gets cited still depends on whether your page answers the question better than the others, and that is something to measure over many runs, not one.
