AI Visibility

How to Check AI Crawler Access: GPTBot, ClaudeBot and More

To check AI crawler access, open your robots.txt and find the rules for each AI user agent, such as GPTBot, OAI-SearchBot and ClaudeBot. Then test key URLs for a 200 response, review firewall and CDN bot settings, and confirm real visits in your server logs. Training and search crawlers are separate, so you can block one and allow the other.
Cihan Geyik
Cihan Geyik
11 min read
Last Updated:
October 14, 2026
How to Check AI Crawler Access: GPTBot, ClaudeBot and More
Explore with AI
Summarize with ChatGPTSummarize with ClaudeSummarize with Google AI
In This Article

OpenAI runs three bots, and each one does a different job. Anthropic runs three more. Google and Apple handle AI training through separate control tokens. So a single "allow AI" or "block AI" rule rarely does what you expect.

You need to know which name does what before you touch robots.txt. We cover each user agent first, then seven checks you can run yourself. Every robots.txt example was checked against the vendor’s documentation in October 2026.

Key takeaways

  • Training crawlers such as GPTBot and ClaudeBot use different robots.txt names than search crawlers such as OAI-SearchBot and Claude-SearchBot.
  • You can block AI training and still appear in ChatGPT search, Claude search and Perplexity, because each bot follows its own rules.
  • Google-Extended and Applebot-Extended are control tokens, not crawlers. Blocking them does not remove you from Google Search or Apple search features.
  • A bot with its own group in robots.txt ignores the User-agent: * group, so repeat any Disallow rules you still want it to follow.
  • User-initiated fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt, because a person asked for the page.
  • A firewall or CDN rule can block an AI crawler that robots.txt allows, so only server or CDN logs prove real access.

What is an AI crawler?

An AI crawler is an automated user agent that fetches web pages for an AI company. The pages may train a model, feed an AI search index, or answer a question a user just asked.

Each crawler identifies itself with a name, such as GPTBot or ClaudeBot. Your robots.txt file uses those names to allow or block each one.

Training crawlers vs search crawlers vs user fetchers

AI bots fall into three groups. The group tells you what you gain or lose when you block a bot.

  • Training crawlers collect pages that may be used to train AI models. Examples are GPTBot and ClaudeBot.
  • Search crawlers build the index that AI answers draw on. Examples are OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and Applebot.
  • User fetchers open a page when a person asks the assistant about it. Examples are ChatGPT-User, Claude-User and Perplexity-User.

Blocking a training crawler won’t remove you from AI search answers. Blocking a search crawler or a user fetcher can.

Google-Extended and Applebot-Extended are a different case. Neither has a crawler of its own. They only tell Google and Apple how pages crawled by Googlebot or Applebot may be used.

Most brands should allow search crawlers and user fetchers. Whether to allow training is a business and legal call, and it has nothing to do with visibility.

Which AI crawler user agents should you check?

These user agents matter for ChatGPT, Claude, Google AI Overviews, AI Mode, Gemini, Microsoft Copilot, Perplexity and Apple. Each behavior below comes from the vendor’s own documentation.

OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User

  • GPTBot crawls content that may be used to train OpenAI’s foundation models. Disallowing it signals that your content should not be used for training.
  • OAI-SearchBot surfaces websites in ChatGPT search. Sites that opt out are not shown in ChatGPT search answers, though they can still appear as navigational links.
  • ChatGPT-User visits pages for user actions in ChatGPT and custom GPTs. OpenAI says robots.txt rules may not apply, because a user started the action.

OpenAI’s documentation on its crawlers says robots.txt changes can take about 24 hours to reach search results. OpenAI also publishes IP address lists for each bot.

Anthropic: ClaudeBot, Claude-SearchBot and Claude-User

  • ClaudeBot collects web content that could contribute to training Anthropic’s models. Blocking it excludes your future pages from training data.
  • Claude-SearchBot indexes content to improve Claude’s search answers. Blocking it may reduce your visibility in Claude’s search results.
  • Claude-User fetches pages when a person asks Claude a question. Blocking it stops Claude from retrieving your content for that user.

Anthropic’s help article on its bots says they honor robots.txt and support the non-standard Crawl-delay rule. It advises against blocking IP addresses, because that stops the bots from reading robots.txt.

Anthropic also asks you to add rules on every subdomain, since each subdomain has its own robots.txt file.

Google: Googlebot and Google-Extended

  • Googlebot crawls for Google Search, which includes AI Overviews and AI Mode. If you block Googlebot, you leave both features and classic results.
  • Google-Extended is a control token for Gemini model training and for grounding in Gemini Apps and Vertex AI. It has no separate user-agent string.

Google’s list of common crawlers says Google-Extended does not affect inclusion or ranking in Google Search. Blocking it can still limit how Gemini Apps use your pages for grounding.

To leave AI Overviews and AI Mode without leaving Search, use the Search generative AI control in Search Console settings. Google’s Search Console Help says it rolled out to all sites on August 31, 2026.

Microsoft: Bingbot

Bingbot is Bing’s standard crawler, described in Bing’s crawler documentation. Microsoft reports Copilot citations in Bing Webmaster Tools, so Bingbot access is the first check for Copilot.

For finer control, Microsoft announced NOCACHE and NOARCHIVE robots meta values in 2023 for Bing Chat, now Microsoft Copilot. It said NOARCHIVE pages are left out of those answers and out of model training.

Perplexity: PerplexityBot and Perplexity-User

  • PerplexityBot surfaces and links websites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models.
  • Perplexity-User visits pages for user actions inside Perplexity. Perplexity says it generally ignores robots.txt, because a user requested the fetch.

Perplexity’s crawler documentation says changes may take up to 24 hours to apply. It also publishes IP lists for both agents.

Apple: Applebot and Applebot-Extended

  • Applebot crawls for Apple features such as Siri and Spotlight. Apple says the data may also be used to train its foundation models.
  • Applebot-Extended does not crawl. Disallowing it opts your content out of Apple’s model training, and your pages can still appear in Apple search results.

One detail catches teams out. Apple’s Applebot page says that if robots.txt names Googlebot but not Applebot, Applebot follows the Googlebot rules.

How to check AI crawler access, step by step

Step 1: Read your robots.txt for each AI user agent

Open https://www.yoursite.com/robots.txt in a browser. Repeat this for every subdomain that hosts content, such as a blog or docs subdomain.

For each bot in the list above, find the group it obeys. A bot follows the group that names it, and only falls back to User-agent: * when no group names it.

This rule comes from the robots.txt standard, RFC 9309, and it causes a lot of accidental gaps. Add a group for GPTBot, and GPTBot stops reading the Disallow lines in your User-agent: * group.

Step 2: Check page-level directives

Robots.txt controls crawling, but page tags control what can be shown. Check the HTML head and the HTTP headers of the pages that matter most.

  • A noindex robots meta tag or X-Robots-Tag header keeps the page out of search, including AI Overviews and AI Mode.
  • nosnippet, data-nosnippet and max-snippet limit what Google can show from the page in its AI features.
  • NOCACHE and NOARCHIVE limit how Bing uses the page in Copilot answers.
  • Templates from staging sites or SEO plugins sometimes add noindex to whole sections without anyone noticing.

Step 3: Test the response each bot receives

Request an important page with an AI user agent and read the status code. You can run this from any terminal:

curl -I -A "GPTBot" https://www.yoursite.com/your-page

A 200 status means the server returned the page. A 403, 429, 503 or a challenge page means something between the bot and your content is refusing it.

This test copies the user-agent name only, not the bot’s IP address. So it catches rules that match the name and misses rules that check IP ranges.

Step 4: Review CDN, firewall and bot-protection rules

Plenty of blocks happen outside robots.txt. CDNs, web application firewalls and bot managers can challenge or reject AI crawlers that robots.txt allows.

Some CDNs, such as Cloudflare, offer one-click settings to block AI crawlers. Look for those toggles in your security dashboard, along with rate limits and any country or IP rules.

Don’t open your firewall to anything that claims to be a known bot. User-agent strings are easy to fake. Allow bots by verified IP lists or through your provider’s verified-bot feature.

Step 5: Make sure key content is in the HTML

Not every AI crawler is documented to run JavaScript. If your main copy or pricing loads only after scripts run, a crawler may get an empty page.

View the page source, not the rendered page, and search for a sentence from your main content. If it’s missing, server-side rendering or static HTML fixes the problem.

Step 6: Confirm real visits in your server or CDN logs

Configuration shows what should happen. Logs show what does happen.

  • Filter access logs by user agent: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Bingbot and Applebot.
  • Note the URLs requested, the status codes returned and how often each bot visits.
  • Check the IP addresses against the lists OpenAI and Perplexity publish, or use Bing’s Verify Bingbot tool.
  • Compare a week of AI bot visits with your sitemap. Look for important sections that no AI crawler reaches.

If an AI crawler never shows up in your logs, check robots.txt and firewall rules first, then internal links. A page that nothing links to is hard for any crawler to find.

Step 7: Use Search Console and Bing Webmaster Tools

Google Search Console has a robots.txt report and a URL Inspection tool that shows whether Googlebot can fetch and index a page. Bing Webmaster Tools has a robots.txt tester and URL inspection.

Both now report AI visibility as well. Search Console shows impressions in AI Overviews and AI Mode. Bing Webmaster Tools shows citations in Microsoft Copilot.

Robots.txt examples for AI crawlers

Treat these templates as a starting point. Each line below is one directive, and each User-agent line starts a new group in your file.

Example 1: Allow AI search, block AI training

This policy opts out of model training wherever a vendor offers a separate control. AI search crawlers still get in, and so do the classic Google and Bing crawlers.

User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Disallow: /admin/

OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and Applebot have no group of their own here. They follow the User-agent: * group and can crawl everything except /admin/.

Mind the Gemini trade-off. Blocking Google-Extended also limits grounding in Gemini Apps. If Gemini visibility matters more to you, skip that group.

Example 2: Allow every crawler

If you want maximum reach in AI answers and accept training use, one group is enough.

User-agent: *
Allow: /

Example 3: Give one bot its own rules without opening private paths

When you add a named group, copy every Disallow line from the User-agent: * group into it. Skip that step and the named bot can crawl paths you meant to keep private.

User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/
Disallow: /checkout/

Worried about crawl volume? Anthropic also supports a Crawl-delay line, such as Crawl-delay: 1 under User-agent: ClaudeBot.

Common reasons AI crawlers cannot access a website

  • A blanket User-agent: * group with Disallow: / blocks every bot that lacks its own group.
  • A rule meant to stop training bots also names search bots such as OAI-SearchBot.
  • The homepage is open, but /blog/ or /docs/ is disallowed.
  • The firewall answers bots with a 403 or 429, or with a browser check they can’t pass.
  • The HTML response is a JavaScript shell with no main text in it.
  • A stray noindex from staging settings or a plugin hides pages from search and its AI features.

After you check AI crawler access, measure AI visibility

Allowing a crawler makes your pages eligible, and that’s all it does. No AI answer is obliged to mention or cite you.

Once access is fixed, track whether your pages get cited for the prompts that matter. Our AI visibility audit checklist covers the content and citation checks that come next, including brand signals.

If your pages are crawlable but still missing, read why a brand doesn’t appear in ChatGPT or Google AI answers.

How Ansvisor helps you monitor AI crawler access

Ansvisor, an open-source AI search intelligence platform, puts AI crawler activity and AI referral visits side by side in AI Traffic Analytics.

The AI Visibility Site Audit checks robots.txt, AI bot access and canonical URLs as part of 47 weighted AEO and GEO signals for any public URL.

Once bots can reach your pages, Ansvisor tracks whether ChatGPT, Gemini, AI Overviews, AI Mode and Copilot cite them. For the referral side, read how to track traffic from ChatGPT and AI search.

Start a 14-day free trial to check your crawler access and AI visibility, or self-host the open-source project on GitHub.

FAQ

How do I check which AI bots can access my website?

Open yoursite.com/robots.txt and note which group each AI user agent follows. Then request an important page with that user agent and check for a 200 response. Finally, look for verified bot visits in your server or CDN logs.

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI uses GPTBot for model training and OAI-SearchBot for ChatGPT search. Blocking GPTBot opts you out of training. Blocking OAI-SearchBot is what keeps your pages out of ChatGPT search answers.

Does blocking Google-Extended remove my site from AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search, which includes AI Overviews and AI Mode. It controls Gemini training and grounding in Gemini Apps and Vertex AI. To leave AI Overviews and AI Mode, Google offers a separate setting in Search Console.

Is my site being crawled by AI engines at all?

Check your server or CDN logs for AI user agents such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. Confirm the IP addresses against vendor lists, since user agents can be faked. If you see no visits for weeks, check robots.txt and firewall rules first, then internal links.

Why does a tool say crawlers are not allowed to access this page?

The message usually means a rule blocks automated access to that URL. Check robots.txt for a Disallow line that matches the bot and path. Then look for noindex tags, and for firewall rules that return 403 errors.

Crawler access is an eligibility layer, not the outcome. First make sure the systems you care about can access your important content. Then measure whether your brand is actually appearing across relevant prompts, citations, competitors, and AI Search experiences.
— Cihan Geyik, Co-founder of Ansvisor
About the Author
Cihan Geyik

Cihan Geyik

Co-founder at Ansvisor

Cihan Geyik is the co-founder of Ansvisor, an open-source, cloud-ready AI Visibility platform for AI Search. With more than 15 years of experience in digital marketing and growth, he writes about AI visibility, AI search, AEO, GEO, citations, and answer engines. He focuses on helping brands understand and improve their presence across ChatGPT, Gemini, Perplexity, Google AI Overviews, and other AI-powered discovery platforms.

← Explore All

Related Blog Posts

Let AI Search Work for Your Growth

Each prompt you win brings more mentions, more citations and more buyers. Ansvisor keeps finding the next one.

AI Answers Mentions Citations Visibility Score AI Share of Voice AI Traffic Revenue
Win Back Search VisibilityTry Ansvisor 14-day Free →✓ Zero risk. Cancel anytime.
Google Preferred Sources
Get more Ansvisor insights in Google

Add Ansvisor as a preferred source and Google will show you more of our AI search, AEO and GEO guides.