GPTBot (OpenAI Web Crawler)

GPTBot is OpenAI's web crawler for content that may be used to train its generative AI foundation models. OpenAI states that GPTBot is used to help make these models more useful and safe.

Website owners can manage GPTBot through robots.txt. OpenAI documents GPTBot separately from OAI-SearchBot, allowing publishers to make different decisions about model-training crawling and ChatGPT Search crawling.

What Does GPTBot Do?

GPTBot automatically crawls web content on behalf of OpenAI. OpenAI describes GPTBot as a crawler used to make its generative AI foundation models more useful and safe.

Content crawled by GPTBot may be used in training those foundation models. Website owners that do not want their content crawled for this purpose can communicate that preference by disallowing GPTBot.

How Does GPTBot Work?

GPTBot requests publicly accessible web resources using its documented crawler identity.

A simplified access path looks like this:

The site's Robots Exclusion Protocol (REP) rules can communicate whether GPTBot is permitted to crawl specific paths.

Technical infrastructure can also affect whether the crawler can actually retrieve a resource, including servers, CDNs, firewalls, bot-management systems, authentication, and other access controls.

How to Allow GPTBot in Robots.txt

A publisher that wants to permit GPTBot to crawl its public website can use:

More granular rules can be used when a site wants to allow some sections while excluding others.

The applicable rule structure follows robots.txt conventions and allows website owners to express crawler-specific preferences.

How to Block GPTBot

A website owner who does not want GPTBot to crawl the site can use:

OpenAI states that disallowing GPTBot indicates that the site's content should not be used in training its generative AI foundation models.

GPTBot vs. OAI-SearchBot

One of the most important distinctions for website owners is the difference between GPTBot and OAI-SearchBot.

OpenAI provides independent robots.txt controls for these two crawlers.

This separation means a publisher does not need to make one universal "allow OpenAI" or "block OpenAI" decision.

Can You Block GPTBot but Allow ChatGPT Search?

Yes. OpenAI explicitly documents GPTBot and OAI-SearchBot as independent controls.

A website can therefore use:

This configuration allows OpenAI's search crawler while indicating that the website's content should not be crawled by GPTBot for use in training OpenAI's generative AI foundation models.

For a detailed explanation of the search crawler, see OAI-SearchBot.

Does Blocking GPTBot Block ChatGPT Search?

No. GPTBot is not the robots.txt control OpenAI provides for managing ChatGPT Search crawling.

OpenAI instructs website owners to use OAI-SearchBot when managing whether their content can participate in automatic Search crawling.

This means blocking GPTBot alone should not be interpreted as blocking OAI-SearchBot.

GPTBot vs. ChatGPT-User

GPTBot also differs from ChatGPT-User.

ChatGPT-User is used for certain actions initiated by users in ChatGPT and Custom GPTs. OpenAI states that it is not used for automatic web crawling.

Because ChatGPT-User represents user-triggered activity rather than automatic web crawling, OpenAI notes that robots.txt rules may not apply to those requests in the same way.

Does GPTBot Train ChatGPT Directly?

It is more accurate to use OpenAI's documented terminology than to say that GPTBot simply "trains ChatGPT."

OpenAI describes GPTBot as crawling content that may be used in training its generative AI foundation models.

Crawling is also not the same thing as training. GPTBot performs the crawling stage; content it crawls may subsequently be used within OpenAI's model-training processes.

Does GPTBot Crawl Every Website?

No. A website can communicate crawling restrictions through robots.txt, and technical access restrictions can also prevent a crawler from retrieving content.

Whether GPTBot successfully accesses a particular URL can therefore depend on both crawler policy and actual technical accessibility.

This relationship is covered more broadly by AI Crawler Accessibility.

GPTBot and Crawler Access

GPTBot access has two distinct layers:

1. Policy access: Does robots.txt permit GPTBot to crawl the requested path?
2. Technical access: Can GPTBot successfully retrieve the resource through the website's infrastructure?

A crawler can be allowed by robots.txt but still fail to retrieve content because of infrastructure-level restrictions.

GPTBot and Firewalls, CDNs, and Bot Protection

Modern websites often place multiple infrastructure layers between a crawler and the origin server.

These can include:

• content delivery networks;
• web application firewalls;
• bot-management systems;
• DDoS protection;
• CAPTCHA systems;
• JavaScript challenges;
• authentication;
• rate limiting;
• IP restrictions; and
• geographic access controls.

Website owners should therefore distinguish between a robots.txt preference and actual network-level access.

GPTBot IP Addresses

OpenAI publishes IP ranges associated with GPTBot. The maintained list is available from:

OpenAI GPTBot IP ranges.

Because crawler infrastructure can change, technical teams should use OpenAI's maintained data rather than relying on a static IP list copied into documentation.

GPTBot User-Agent

GPTBot identifies itself through an HTTP user-agent containing the GPTBot token.

OpenAI's crawler documentation notes that the version number shown in its example user-agent string may change.

When retrieving robots.txt, OpenAI may also include a robots.txt marker in the user-agent string to help site owners distinguish those requests in server logs.

GPTBot and AI Search Indexing

GPTBot should not automatically be described as an AI Search Indexing crawler.

OpenAI specifically provides OAI-SearchBot for Search. GPTBot's documented role concerns content that may be used in training OpenAI's generative AI foundation models.

Keeping these concepts separate prevents technical teams from treating model training, search discovery, retrieval, and answer generation as one identical process.

Does Allowing GPTBot Improve AI Visibility?

There is no basis for treating GPTBot access as a direct AI Visibility ranking signal.

Allowing GPTBot concerns whether OpenAI can crawl content for potential use in generative AI foundation model training. It should not be confused with allowing OAI-SearchBot for ChatGPT Search.

Does Blocking GPTBot Reduce ChatGPT Citations?

Blocking GPTBot should not automatically be interpreted as removing a website from ChatGPT Search citations.

Search crawling is controlled separately through OAI-SearchBot. Consequently, teams evaluating AI Citations should distinguish training-related crawler policy from search-related crawler accessibility.

Should Websites Allow GPTBot?

There is no universal answer.

The decision depends on the publisher's policy regarding use of its content for generative AI foundation model training.

Organizations may choose to:

• allow GPTBot across the entire public website;
• block GPTBot across the entire website;
• allow selected public sections while restricting others; or
• apply a different policy to GPTBot and OAI-SearchBot.

The important technical principle is to make this decision deliberately rather than assuming all OpenAI crawlers perform the same function.

Common GPTBot Mistakes

• confusing GPTBot with OAI-SearchBot;
• assuming GPTBot is required for ChatGPT Search visibility;
• assuming blocking GPTBot automatically blocks ChatGPT Search;
• treating ChatGPT-User as another name for GPTBot;
• describing crawling and model training as the same process;
• assuming GPTBot access is an AI Search ranking factor;
• assuming allowing GPTBot guarantees AI citations;
• using one generic rule without considering separate OpenAI crawler purposes; and
• relying on outdated crawler identity or IP information.

How to Audit GPTBot Access

1. Decide your training policy. Determine whether the organization wants GPTBot to crawl public content for potential model-training use.
2. Review robots.txt. Identify explicit GPTBot rules and broader user-agent rules that may affect it.
3. Review path-level rules. Confirm whether important public and restricted sections reflect the intended policy.
4. Separate GPTBot from OAI-SearchBot. Make independent decisions for model-training crawling and search crawling.
5. Review infrastructure. If GPTBot is intended to be accessible, check whether security systems accidentally prevent access.
6. Use current OpenAI documentation. Crawler identities, versions, and infrastructure can change.

GPTBot in an AI Search Technical Strategy

GPTBot belongs in an AI crawler governance strategy, but it should not be treated as a direct AI Search optimization mechanism.

A useful technical framework separates:

This separation helps organizations make informed crawler decisions while avoiding the assumption that allowing every AI crawler automatically improves search performance.

From Crawler Policy to AI Search Measurement

Crawler configuration is only one part of an AI Search strategy. Search performance still needs to be measured independently.

Ansvisor's ChatGPT Visibility Tracker can be used to monitor actual brand visibility across relevant ChatGPT prompts.

With Ansvisor's Citation Monitoring, teams can also measure which domains and URLs are actually appearing as sources rather than inferring search performance from crawler policy.

The broader Ansvisor AI Search Intelligence Platform connects this measurement with prompts, competitors, sources, opportunities, and actions.