
When ChatGPT searches the web, the pages it can show come from crawling by OAI-SearchBot. Block that one bot and your content drops out of ChatGPT search answers. It’s easy to do by accident, through a wildcard rule in robots.txt or a firewall setting aimed at scrapers. Knowing exactly what this crawler does keeps your training choices and your search visibility apart.
OAI-SearchBot is the crawler OpenAI uses for search. According to OpenAI’s crawler documentation, it surfaces websites in ChatGPT’s search features.
You control it through robots.txt with the token OAI-SearchBot. OpenAI recommends allowing the bot, and its published IP ranges, if you want your site to appear in search results.
It helps to know what the bot doesn’t do. It doesn’t gather training data, which is GPTBot’s job. It also isn’t the agent that opens a page during a chat because a user asked for it. That’s ChatGPT-User. Mixing them up causes most of the mistakes listed further down.
Changes aren’t instant. OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust.
| Agent | Purpose | Robots.txt |
|---|---|---|
| OAI-SearchBot | Crawls pages to surface them in ChatGPT search | Applies |
| GPTBot | Crawls content that may train OpenAI’s foundation models | Applies |
| ChatGPT-User | Visits a page when a user action in ChatGPT needs it | May not apply, since a user starts the request |
| OAI-AdsBot | Checks the safety of pages submitted as ChatGPT ads | Visits submitted ad pages only |
OpenAI states that each setting is independent of the others. One detail is easy to miss. If you allow both OAI-SearchBot and GPTBot, OpenAI may use one crawl for both purposes to avoid duplicate requests.
ChatGPT-User isn’t used to decide whether content can appear in search. For a search opt-out, OAI-SearchBot is the control that counts.
Add a group for the bot to your robots.txt file:
To stay in ChatGPT search while opting out of model training, pair it with a GPTBot block:
You rarely need a full block. If some areas shouldn’t appear in ChatGPT, such as account pages or internal search results, close just those paths:
Check the wildcard group too. A User-agent: * group with Disallow: / blocks OAI-SearchBot unless the file also has a group that names it.
The full user-agent string looks like this. OpenAI notes that the version number may change, so match on the OAI-SearchBot token.
Anyone can copy that string. To confirm a request is real, compare its IP with the OAI-SearchBot IP ranges that OpenAI publishes. Use the same file to build firewall allow rules. GPTBot and ChatGPT-User each have their own IP file.
Robots.txt is the first gate, not the last. A CDN rule, a web application firewall or a JavaScript challenge can turn OAI-SearchBot away even when the file allows it.
Search your server logs for 403 and 429 responses to OpenAI’s IP ranges. Then fetch a few priority pages with the bot’s user agent and compare the result with what a browser gets. The same checks apply to other AI crawlers.
Only in a narrow way. OpenAI says opted-out sites won’t be shown in ChatGPT search answers, but they can still appear as navigational links.
So a block doesn’t make your URL unknown to ChatGPT. It keeps your content out of the answers, which is the opposite of what a brand chasing visibility wants.
It makes you eligible. Each answer still picks a handful of sources, and ChatGPT picks far fewer than Gemini or Google AI Overviews.
In Ansvisor tracking data, September 15 to October 14, 2026, ChatGPT answers carried about 5.2 links each. Gemini answers carried about 12.3. With fewer slots per answer, the page that wins is the one that answers the question best.
Our guide on getting cited in ChatGPT search covers that work. For how the product picks and displays sources, see ChatGPT search.
OpenAI’s publisher FAQ says ChatGPT adds utm_source=chatgpt.com to referral URLs. That makes ChatGPT visits easy to separate in analytics tools such as GA4.
Some visits still arrive without the tag or a referrer, for example when someone copies a link. Treat the tagged number as a floor. The wider practice is called AI traffic analytics.
Repeat the check after any security change or site migration. Either one can change bot access without anyone noticing.
Ansvisor’s Site Audit checks robots.txt and AI bot access, so a blocked OAI-SearchBot shows up before it costs you answers. The ChatGPT visibility tracker then shows which prompts mention or cite your brand and which of your URLs ChatGPT uses. AI Traffic Analytics adds the visits ChatGPT sends to each page.
Each prompt you win brings more mentions, more citations and more buyers. Ansvisor keeps finding the next one.
OAI-SearchBot is OpenAI’s crawler for search. It surfaces websites in ChatGPT’s search features. OpenAI says sites that opt out of it won’t be shown in ChatGPT search answers, though they can still appear as navigational links.
Add a group with User-agent: OAI-SearchBot and Allow: / to your robots.txt file. OpenAI also recommends allowing its published IP ranges in your firewall. Changes can take about 24 hours to reach OpenAI’s systems.
OAI-SearchBot crawls for ChatGPT search. GPTBot crawls content that may be used to train OpenAI’s foundation models. The settings are independent, so you can allow OAI-SearchBot and block GPTBot in the same file.
OAI-SearchBot crawls automatically for search. ChatGPT-User visits a page when a user’s request in ChatGPT or a Custom GPT needs it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and it isn’t used to decide search inclusion.
No. Allowing it makes your pages eligible for ChatGPT search answers. Whether a page gets cited depends on the question and the competing sources, so track real answers to see which of your URLs ChatGPT uses.
See how AI answers describe your brand, find out why competitors get picked, and close the gaps with AI agents that work alongside your team.
14-day free trial or 50% off your first month with code SAVE50.
Continue exploring key AI visibility concepts.
Measure and improve how often your brand appears in AI-generated answers.
Learn more →Strategies for increasing visibility in answer engines and AI summaries.
Learn more →Optimizing content for AI-powered discovery experiences.
Learn more →Understand how OpenAI retrieves and synthesizes information.
Learn more →AI-generated summaries that appear directly in Google Search.
Learn more →Explore how Perplexity cites and presents sources.
Learn more →References and sources used by AI systems to support answers.
Learn more →Measure the quality and influence of cited sources.
Learn more →How easily AI systems can discover and reuse your content.
Learn more →New terms are added regularly.
Help us improve the page or suggest a new term →
Co-founder at Ansvisor
Cihan Geyik is the co-founder of Ansvisor, an open-source AI Visibility platform for AI Search. With more than 15 years of experience in digital marketing and growth, he writes about AI visibility, AI search, AEO, GEO, citations, and answer engines. He focuses on helping brands understand and improve their presence across ChatGPT, Gemini, Google AI Overviews, Bing AI Mode and other AI answer engines.
Add Ansvisor as a preferred source and Google will show you more of our AI search, AEO and GEO guides.
© 2026 Ansvisor. All rights reserved. Ansvisor is an open-source AI Search Execution Platform for AI Visibility. Ansvisor is a brand owned and operated by Empler AI Inc.
The open-source AI Search Execution Platform. Track how AI answers talk about your brand and act on it.
Open source · MIT license© 2026 Ansvisor, a brand of Empler AI Inc. · 1401 PE Ave. Unit 105, Wilmington, DE 19806, United States
Cookie preferences
We use cookies to keep the site working and, with your OK, to measure traffic and improve our marketing. Read our Cookies & Tracking policy.