AI & Infrastructure
GPTBot (OpenAI Web Crawler)

GPTBot (OpenAI Web Crawler)

GPTBot is OpenAI’s web crawler for content that may be used to train its generative AI foundation models. Site owners allow or block it in robots.txt, separately from OpenAI’s search crawler.
October 5, 2026
Cihan Geyik
Table of Content

Most sites that block GPTBot want one outcome: keep their content out of OpenAI’s model training without vanishing from ChatGPT search. That works, because OpenAI runs a separate crawler for each job. GPTBot is the training crawler. It doesn’t decide whether ChatGPT search can show or cite your pages, so blocking it is a narrower decision than many teams assume.

Blocking GPTBot is a training decision. OpenAI says a GPTBot disallow signals that your content shouldn’t be used to train its generative AI foundation models. Whether ChatGPT search can cite you depends on OAI-SearchBot.

What is GPTBot?

GPTBot is the web crawler OpenAI uses to collect content for its foundation models. OpenAI’s crawler documentation says GPTBot is used to make those models more useful and safe. Content it crawls may be used in training.

It’s one of several AI crawlers that now visit most public sites. Like the others, it reads robots.txt, follows the group that applies to it and requests public URLs. Pages behind a login stay out of its reach.

A block has limits, though. It applies to future crawling, and OpenAI’s documentation doesn’t describe removing content collected before the block. Your text can also reach a model indirectly, when other sites quote or summarize it.

GPTBot user agent and IP ranges

OpenAI lists this user agent string for GPTBot. The version number can change, so match on the GPTBot token in rules and log filters.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

When GPTBot fetches robots.txt itself, OpenAI may add a robots.txt marker to the string. The marker identifies those requests even in logs that don’t record the path.

Anyone can copy the string. To confirm a request comes from GPTBot, check its IP against the GPTBot IP ranges that OpenAI publishes. A quick first pass on a server log:

grep "GPTBot" access.log | cut -d" " -f1 | sort | uniq -c | sort -rn

It lists each requesting IP with a hit count. Hits from outside the published ranges aren’t OpenAI, whatever the user agent says.

Does GPTBot respect robots.txt?

OpenAI presents robots.txt as the way to manage GPTBot, and a disallow is how you record a training opt-out. Its documentation says robots.txt rules may not apply to ChatGPT-User, the agent behind user-initiated visits. It makes no such exception for GPTBot.

If you see GPTBot-labeled requests on paths you’ve disallowed, check the IPs before drawing conclusions. Spoofed user agents are a common source of "rule-breaking" bot traffic in logs.

How to block GPTBot in robots.txt

Add a group for GPTBot to your robots.txt file. This opts the whole site out of GPTBot crawling:

User-agent: GPTBot
Disallow: /

You can also block part of a site. A publisher might keep its archive and paid articles out of training and leave everything else open:

User-agent: GPTBot
Disallow: /archive/
Disallow: /premium/
Allow: /

Watch the wildcard group. Under RFC 9309, a crawler with no group of its own follows User-agent: *. If that group says Disallow: /, GPTBot is blocked even though no line names it.

Each host needs its own file, too. A block on www.example.com doesn’t cover docs.example.com.

How to allow GPTBot

If you’re happy for your public content to inform OpenAI’s models, name the bot and allow it:

User-agent: GPTBot
Allow: /

An explicit group matters when your wildcard rules are strict. It also documents the decision for whoever edits the file next. Then confirm your CDN or firewall isn’t blocking OpenAI’s IP ranges, or the allow rule does nothing.

GPTBot vs OAI-SearchBot vs ChatGPT-User

AgentWhat it doesrobots.txt
GPTBotCollects content that may train OpenAI’s foundation modelsYour control for training use
OAI-SearchBotCrawls pages so ChatGPT search can show and cite themYour control for search inclusion
ChatGPT-UserVisits a page when a user action in ChatGPT needs itMay not apply, per OpenAI

OpenAI states that each setting is independent of the others. Our OAI-SearchBot page covers the search crawler in depth, including what happens to sites that opt out of it.

Can you block GPTBot and stay in ChatGPT search?

Yes. It’s the setup most brands ask about:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

OAI-SearchBot keeps crawling for search, and the GPTBot group records your training opt-out. OpenAI adds one detail. If a site allows both bots, it may use the results of a single crawl for both purposes.

Other AI companies split their bots the same way. Anthropic separates ClaudeBot from Claude-SearchBot. Google uses the Google-Extended token for Gemini training and grounding, while Googlebot keeps crawling for Search.

Should you allow GPTBot?

No setting suits every site. The real question is whether you want OpenAI’s future models to learn from your pages.

  • Publishers that license content often block GPTBot on paid or archived sections. Their text is the product.
  • SaaS and ecommerce brands usually allow it. Their pages describe products they want models to know about.
  • Docs teams tend to allow it, so models answer setup questions with current instructions.
  • Sites with legal limits on reuse block it, and put anything truly private behind authentication.

Either choice has a cost. When ChatGPT answers from model memory without a live search, it draws on what it learned in training. Block GPTBot, and those answers can lean more on what other sites say about you.

Allowing GPTBot won’t earn you citations. It feeds training data. ChatGPT search picks its sources at answer time, and OAI-SearchBot is the crawler you control for that.

Common GPTBot mistakes

  • Blocking GPTBot and expecting to disappear from ChatGPT search. Search inclusion runs through OAI-SearchBot.
  • Allowing GPTBot and expecting more citations. OpenAI doesn’t describe it as a ranking or citation signal.
  • Forgetting the wildcard group, which quietly covers GPTBot when it has no group of its own.
  • Trusting the user agent string without checking IPs against gptbot.json.
  • Copying a bot list from an old blog post. Version numbers change, and OpenAI updates its documentation.
  • Fixing robots.txt on the main domain and missing subdomains.

How to audit GPTBot access

Decide your training policy → Read robots.txt on every host → Check CDN and firewall rules → Verify GPTBot IPs in logs → Recheck after site changes
  1. Agree on a policy with legal and content owners: allow, block, or block selected paths.
  2. Open robots.txt on each host and find the group that applies to GPTBot. If none names it, the wildcard group applies.
  3. Ask whoever runs your CDN or firewall how AI bots are handled. A blanket rule there can override what robots.txt says.
  4. Filter a week of logs by the GPTBot token, match IPs to gptbot.json and review the status codes.
  5. Repeat the check after migrations and security changes. Either one can change bot access without anyone noticing.

The same routine works for every major bot. Our post on checking if AI crawlers can access your website walks through it, and AI crawler accessibility covers the layers that can block a bot.

How Ansvisor helps

Ansvisor’s Site Audit checks robots.txt and AI bot access, so you can confirm GPTBot and OAI-SearchBot get the treatment you chose. Answer tracking then shows whether ChatGPT names and cites your brand across your prompt set, by market and language. Start a 14-day free trial to see where ChatGPT cites you today.

GPTBot, GPT Bot, OpenAI GPTBot, OpenAI Web Crawler, OpenAI Training Crawler, GPTBot User Agent

Let AI Search Work for Your Growth

Each prompt you win brings more mentions, more citations and more buyers. Ansvisor keeps finding the next one.

AI Answers Mentions Citations Visibility Score AI Share of Voice AI Traffic Revenue
Win Back Search VisibilityTry Ansvisor 14-day Free →✓ Zero risk. Cancel anytime.

FAQ

Frequently asked questions.

What is GPTBot?

GPTBot is OpenAI’s web crawler for content that may be used to train its generative AI foundation models. It identifies itself with the GPTBot token, and site owners allow or block it in robots.txt. It is separate from OAI-SearchBot, which crawls for ChatGPT search.

How do I block GPTBot?

Add “User-agent: GPTBot” followed by “Disallow: /” to robots.txt to block the whole site, or disallow only specific paths. OpenAI says a GPTBot disallow signals that your content should not be used for training. Give each subdomain its own rules.

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI says its crawler settings are independent of each other. ChatGPT search inclusion runs through OAI-SearchBot. You can block GPTBot and still allow OAI-SearchBot, which keeps your pages eligible for search answers.

How can I verify that traffic really comes from GPTBot?

Match the request IP against the GPTBot IP ranges OpenAI publishes at openai.com/gptbot.json. The user agent string alone proves nothing, because anyone can copy it. Refresh firewall rules from that file, since the ranges can change.

Does allowing GPTBot improve AI visibility?

Not directly. GPTBot collects training data, and OpenAI doesn’t describe it as a ranking or citation signal. Citations in ChatGPT search depend on search crawling and on how well your pages answer the question, so track them separately.

Help us grow the AI Visibility Glossary

New terms are added regularly.

Help us improve the page or suggest a new term →
About the Author
Cihan Geyik

Cihan Geyik

Co-founder at Ansvisor

Cihan Geyik is the co-founder of Ansvisor, an open-source AI Visibility platform for AI Search. With more than 15 years of experience in digital marketing and growth, he writes about AI visibility, AI search, AEO, GEO, citations, and answer engines. He focuses on helping brands understand and improve their presence across ChatGPT, Gemini, Google AI Overviews, Bing AI Mode and other AI answer engines.

Google Preferred Sources
Get more Ansvisor insights in Google

Add Ansvisor as a preferred source and Google will show you more of our AI search, AEO and GEO guides.