
Most sites that block GPTBot want one outcome: keep their content out of OpenAI’s model training without vanishing from ChatGPT search. That works, because OpenAI runs a separate crawler for each job. GPTBot is the training crawler. It doesn’t decide whether ChatGPT search can show or cite your pages, so blocking it is a narrower decision than many teams assume.
GPTBot is the web crawler OpenAI uses to collect content for its foundation models. OpenAI’s crawler documentation says GPTBot is used to make those models more useful and safe. Content it crawls may be used in training.
It’s one of several AI crawlers that now visit most public sites. Like the others, it reads robots.txt, follows the group that applies to it and requests public URLs. Pages behind a login stay out of its reach.
A block has limits, though. It applies to future crawling, and OpenAI’s documentation doesn’t describe removing content collected before the block. Your text can also reach a model indirectly, when other sites quote or summarize it.
OpenAI lists this user agent string for GPTBot. The version number can change, so match on the GPTBot token in rules and log filters.
When GPTBot fetches robots.txt itself, OpenAI may add a robots.txt marker to the string. The marker identifies those requests even in logs that don’t record the path.
Anyone can copy the string. To confirm a request comes from GPTBot, check its IP against the GPTBot IP ranges that OpenAI publishes. A quick first pass on a server log:
It lists each requesting IP with a hit count. Hits from outside the published ranges aren’t OpenAI, whatever the user agent says.
OpenAI presents robots.txt as the way to manage GPTBot, and a disallow is how you record a training opt-out. Its documentation says robots.txt rules may not apply to ChatGPT-User, the agent behind user-initiated visits. It makes no such exception for GPTBot.
If you see GPTBot-labeled requests on paths you’ve disallowed, check the IPs before drawing conclusions. Spoofed user agents are a common source of "rule-breaking" bot traffic in logs.
Add a group for GPTBot to your robots.txt file. This opts the whole site out of GPTBot crawling:
You can also block part of a site. A publisher might keep its archive and paid articles out of training and leave everything else open:
Watch the wildcard group. Under RFC 9309, a crawler with no group of its own follows User-agent: *. If that group says Disallow: /, GPTBot is blocked even though no line names it.
Each host needs its own file, too. A block on www.example.com doesn’t cover docs.example.com.
If you’re happy for your public content to inform OpenAI’s models, name the bot and allow it:
An explicit group matters when your wildcard rules are strict. It also documents the decision for whoever edits the file next. Then confirm your CDN or firewall isn’t blocking OpenAI’s IP ranges, or the allow rule does nothing.
| Agent | What it does | robots.txt |
|---|---|---|
| GPTBot | Collects content that may train OpenAI’s foundation models | Your control for training use |
| OAI-SearchBot | Crawls pages so ChatGPT search can show and cite them | Your control for search inclusion |
| ChatGPT-User | Visits a page when a user action in ChatGPT needs it | May not apply, per OpenAI |
OpenAI states that each setting is independent of the others. Our OAI-SearchBot page covers the search crawler in depth, including what happens to sites that opt out of it.
Yes. It’s the setup most brands ask about:
OAI-SearchBot keeps crawling for search, and the GPTBot group records your training opt-out. OpenAI adds one detail. If a site allows both bots, it may use the results of a single crawl for both purposes.
Other AI companies split their bots the same way. Anthropic separates ClaudeBot from Claude-SearchBot. Google uses the Google-Extended token for Gemini training and grounding, while Googlebot keeps crawling for Search.
No setting suits every site. The real question is whether you want OpenAI’s future models to learn from your pages.
Either choice has a cost. When ChatGPT answers from model memory without a live search, it draws on what it learned in training. Block GPTBot, and those answers can lean more on what other sites say about you.
The same routine works for every major bot. Our post on checking if AI crawlers can access your website walks through it, and AI crawler accessibility covers the layers that can block a bot.
Ansvisor’s Site Audit checks robots.txt and AI bot access, so you can confirm GPTBot and OAI-SearchBot get the treatment you chose. Answer tracking then shows whether ChatGPT names and cites your brand across your prompt set, by market and language. Start a 14-day free trial to see where ChatGPT cites you today.
Each prompt you win brings more mentions, more citations and more buyers. Ansvisor keeps finding the next one.
GPTBot is OpenAI’s web crawler for content that may be used to train its generative AI foundation models. It identifies itself with the GPTBot token, and site owners allow or block it in robots.txt. It is separate from OAI-SearchBot, which crawls for ChatGPT search.
Add “User-agent: GPTBot” followed by “Disallow: /” to robots.txt to block the whole site, or disallow only specific paths. OpenAI says a GPTBot disallow signals that your content should not be used for training. Give each subdomain its own rules.
No. OpenAI says its crawler settings are independent of each other. ChatGPT search inclusion runs through OAI-SearchBot. You can block GPTBot and still allow OAI-SearchBot, which keeps your pages eligible for search answers.
Match the request IP against the GPTBot IP ranges OpenAI publishes at openai.com/gptbot.json. The user agent string alone proves nothing, because anyone can copy it. Refresh firewall rules from that file, since the ranges can change.
Not directly. GPTBot collects training data, and OpenAI doesn’t describe it as a ranking or citation signal. Citations in ChatGPT search depend on search crawling and on how well your pages answer the question, so track them separately.
See how AI answers describe your brand, find out why competitors get picked, and close the gaps with AI agents that work alongside your team.
14-day free trial or 50% off your first month with code SAVE50.
Continue exploring key AI visibility concepts.
Measure and improve how often your brand appears in AI-generated answers.
Learn more →Strategies for increasing visibility in answer engines and AI summaries.
Learn more →Optimizing content for AI-powered discovery experiences.
Learn more →Understand how OpenAI retrieves and synthesizes information.
Learn more →AI-generated summaries that appear directly in Google Search.
Learn more →Explore how Perplexity cites and presents sources.
Learn more →References and sources used by AI systems to support answers.
Learn more →Measure the quality and influence of cited sources.
Learn more →How easily AI systems can discover and reuse your content.
Learn more →New terms are added regularly.
Help us improve the page or suggest a new term →
Co-founder at Ansvisor
Cihan Geyik is the co-founder of Ansvisor, an open-source AI Visibility platform for AI Search. With more than 15 years of experience in digital marketing and growth, he writes about AI visibility, AI search, AEO, GEO, citations, and answer engines. He focuses on helping brands understand and improve their presence across ChatGPT, Gemini, Google AI Overviews, Bing AI Mode and other AI answer engines.
Add Ansvisor as a preferred source and Google will show you more of our AI search, AEO and GEO guides.
© 2026 Ansvisor. All rights reserved. Ansvisor is an open-source AI Search Execution Platform for AI Visibility. Ansvisor is a brand owned and operated by Empler AI Inc.
The open-source AI Search Execution Platform. Track how AI answers talk about your brand and act on it.
Open source · MIT license© 2026 Ansvisor, a brand of Empler AI Inc. · 1401 PE Ave. Unit 105, Wilmington, DE 19806, United States
Cookie preferences
We use cookies to keep the site working and, with your OK, to measure traffic and improve our marketing. Read our Cookies & Tracking policy.