AI & Infrastructure
OAI-SearchBot (OpenAI Search Crawler)

OAI-SearchBot (OpenAI Search Crawler)

OAI-SearchBot is OpenAI’s web crawler for search. It reads pages so they can be shown and cited in ChatGPT search answers, and site owners control it through robots.txt.
October 5, 2026
Cihan Geyik
Table of Content

When ChatGPT searches the web, the pages it can show come from crawling by OAI-SearchBot. Block that one bot and your content drops out of ChatGPT search answers. It’s easy to do by accident, through a wildcard rule in robots.txt or a firewall setting aimed at scrapers. Knowing exactly what this crawler does keeps your training choices and your search visibility apart.

Opting out of OAI-SearchBot removes you from ChatGPT search answers. Opted-out URLs can still surface as plain links, but their content isn’t shown in the answers.

What is OAI-SearchBot?

OAI-SearchBot is the crawler OpenAI uses for search. According to OpenAI’s crawler documentation, it surfaces websites in ChatGPT’s search features.

You control it through robots.txt with the token OAI-SearchBot. OpenAI recommends allowing the bot, and its published IP ranges, if you want your site to appear in search results.

It helps to know what the bot doesn’t do. It doesn’t gather training data, which is GPTBot’s job. It also isn’t the agent that opens a page during a chat because a user asked for it. That’s ChatGPT-User. Mixing them up causes most of the mistakes listed further down.

Changes aren’t instant. OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust.

OAI-SearchBot vs GPTBot vs ChatGPT-User

AgentPurposeRobots.txt
OAI-SearchBotCrawls pages to surface them in ChatGPT searchApplies
GPTBotCrawls content that may train OpenAI’s foundation modelsApplies
ChatGPT-UserVisits a page when a user action in ChatGPT needs itMay not apply, since a user starts the request
OAI-AdsBotChecks the safety of pages submitted as ChatGPT adsVisits submitted ad pages only

OpenAI states that each setting is independent of the others. One detail is easy to miss. If you allow both OAI-SearchBot and GPTBot, OpenAI may use one crawl for both purposes to avoid duplicate requests.

ChatGPT-User isn’t used to decide whether content can appear in search. For a search opt-out, OAI-SearchBot is the control that counts.

How to allow OAI-SearchBot in robots.txt

Add a group for the bot to your robots.txt file:

User-agent: OAI-SearchBot
Allow: /

To stay in ChatGPT search while opting out of model training, pair it with a GPTBot block:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

You rarely need a full block. If some areas shouldn’t appear in ChatGPT, such as account pages or internal search results, close just those paths:

User-agent: OAI-SearchBot
Allow: /
Disallow: /account/
Disallow: /search

Check the wildcard group too. A User-agent: * group with Disallow: / blocks OAI-SearchBot unless the file also has a group that names it.

OAI-SearchBot user agent and IP ranges

The full user-agent string looks like this. OpenAI notes that the version number may change, so match on the OAI-SearchBot token.

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

Anyone can copy that string. To confirm a request is real, compare its IP with the OAI-SearchBot IP ranges that OpenAI publishes. Use the same file to build firewall allow rules. GPTBot and ChatGPT-User each have their own IP file.

Why an allowed bot can still be blocked

Robots.txt is the first gate, not the last. A CDN rule, a web application firewall or a JavaScript challenge can turn OAI-SearchBot away even when the file allows it.

robots.txt allows OAI-SearchBot → CDN and WAF pass the request → Server returns 200 with the page → Page is eligible for ChatGPT search → An answer may cite it

Search your server logs for 403 and 429 responses to OpenAI’s IP ranges. Then fetch a few priority pages with the bot’s user agent and compare the result with what a browser gets. The same checks apply to other AI crawlers.

Can a blocked site still appear in ChatGPT?

Only in a narrow way. OpenAI says opted-out sites won’t be shown in ChatGPT search answers, but they can still appear as navigational links.

So a block doesn’t make your URL unknown to ChatGPT. It keeps your content out of the answers, which is the opposite of what a brand chasing visibility wants.

Does allowing OAI-SearchBot get you cited?

It makes you eligible. Each answer still picks a handful of sources, and ChatGPT picks far fewer than Gemini or Google AI Overviews.

In Ansvisor tracking data, September 15 to October 14, 2026, ChatGPT answers carried about 5.2 links each. Gemini answers carried about 12.3. With fewer slots per answer, the page that wins is the one that answers the question best.

Eligible isn’t the same as cited. Once the bot can reach your pages, the work moves to content and to the third-party sources ChatGPT trusts.

Our guide on getting cited in ChatGPT search covers that work. For how the product picks and displays sources, see ChatGPT search.

How to measure ChatGPT search traffic

OpenAI’s publisher FAQ says ChatGPT adds utm_source=chatgpt.com to referral URLs. That makes ChatGPT visits easy to separate in analytics tools such as GA4.

Some visits still arrive without the tag or a referrer, for example when someone copies a link. Treat the tagged number as a floor. The wider practice is called AI traffic analytics.

How to audit OAI-SearchBot access in five steps

  1. Open yoursite.com/robots.txt on every host you care about and find the group that applies to OAI-SearchBot. If none names it, the wildcard group applies.
  2. Test your priority URLs against those rules: product pages, pricing, docs, comparison pages and top blog posts.
  3. Ask whoever runs your CDN or firewall whether AI bots are challenged or blocked. Exempt OpenAI’s published ranges if they are.
  4. Filter a week of server logs by the OAI-SearchBot token. Confirm the IPs match searchbot.json and the responses are 200.
  5. Wait a day after any robots.txt change, then check whether your pages start showing up as sources for the prompts you track.

Repeat the check after any security change or site migration. Either one can change bot access without anyone noticing.

Common OAI-SearchBot mistakes

  • Blocking GPTBot and assuming ChatGPT search is blocked too. Search runs on OAI-SearchBot.
  • Blocking OAI-SearchBot to stop model training. That removes you from search answers and leaves GPTBot untouched.
  • Keeping a wildcard Disallow with no OAI-SearchBot group.
  • Allowing the bot in robots.txt while the firewall rejects OpenAI’s IPs.
  • Copying an IP list into a config once and never refreshing it from searchbot.json.

How Ansvisor helps

Ansvisor’s Site Audit checks robots.txt and AI bot access, so a blocked OAI-SearchBot shows up before it costs you answers. The ChatGPT visibility tracker then shows which prompts mention or cite your brand and which of your URLs ChatGPT uses. AI Traffic Analytics adds the visits ChatGPT sends to each page.

OAI-SearchBot, OAI SearchBot, OpenAI SearchBot, OpenAI Search Crawler, ChatGPT Search Crawler, ChatGPT Search Bot

Let AI Search Work for Your Growth

Each prompt you win brings more mentions, more citations and more buyers. Ansvisor keeps finding the next one.

AI Answers Mentions Citations Visibility Score AI Share of Voice AI Traffic Revenue
Win Back Search VisibilityTry Ansvisor 14-day Free →✓ Zero risk. Cancel anytime.

FAQ

Frequently asked questions.

What is OAI-SearchBot?

OAI-SearchBot is OpenAI’s crawler for search. It surfaces websites in ChatGPT’s search features. OpenAI says sites that opt out of it won’t be shown in ChatGPT search answers, though they can still appear as navigational links.

How do I allow OAI-SearchBot in robots.txt?

Add a group with User-agent: OAI-SearchBot and Allow: / to your robots.txt file. OpenAI also recommends allowing its published IP ranges in your firewall. Changes can take about 24 hours to reach OpenAI’s systems.

What is the difference between OAI-SearchBot and GPTBot?

OAI-SearchBot crawls for ChatGPT search. GPTBot crawls content that may be used to train OpenAI’s foundation models. The settings are independent, so you can allow OAI-SearchBot and block GPTBot in the same file.

What is the difference between OAI-SearchBot and ChatGPT-User?

OAI-SearchBot crawls automatically for search. ChatGPT-User visits a page when a user’s request in ChatGPT or a Custom GPT needs it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and it isn’t used to decide search inclusion.

Does allowing OAI-SearchBot guarantee ChatGPT citations?

No. Allowing it makes your pages eligible for ChatGPT search answers. Whether a page gets cited depends on the question and the competing sources, so track real answers to see which of your URLs ChatGPT uses.

Help us grow the AI Visibility Glossary

New terms are added regularly.

Help us improve the page or suggest a new term →
About the Author
Cihan Geyik

Cihan Geyik

Co-founder at Ansvisor

Cihan Geyik is the co-founder of Ansvisor, an open-source AI Visibility platform for AI Search. With more than 15 years of experience in digital marketing and growth, he writes about AI visibility, AI search, AEO, GEO, citations, and answer engines. He focuses on helping brands understand and improve their presence across ChatGPT, Gemini, Google AI Overviews, Bing AI Mode and other AI answer engines.

Google Preferred Sources
Get more Ansvisor insights in Google

Add Ansvisor as a preferred source and Google will show you more of our AI search, AEO and GEO guides.