AI & Infrastructure
ClaudeBot / Claude-SearchBot (Anthropic Crawlers)

ClaudeBot / Claude-SearchBot (Anthropic Crawlers)

ClaudeBot and Claude-SearchBot are Anthropic’s web crawlers. ClaudeBot collects public pages that may be used to train Claude models, while Claude-SearchBot crawls pages to improve Claude’s search results.
October 5, 2026
Cihan Geyik
Table of Content

If you read your server logs, you’ve probably seen ClaudeBot. Anthropic runs three web agents, and each one maps to a separate decision. Should your content train Claude models? Should it appear in Claude’s search results? Should Claude fetch your page when a user asks about it? You can say yes to one and no to another.

Training and search are separate switches. Blocking ClaudeBot opts you out of training. It doesn’t remove you from Claude search, which runs on Claude-SearchBot and Claude-User.

What is ClaudeBot?

ClaudeBot is Anthropic’s crawler for model development. It collects public web content that could contribute to training Anthropic’s generative AI models.

A block has a specific meaning. Anthropic’s crawler documentation says a ClaudeBot restriction signals that the site’s future materials should be excluded from training. Note the word future. The opt-out works going forward.

What is Claude-SearchBot?

Claude-SearchBot crawls the web to improve search results for Claude users. It reads pages so Claude can return more relevant and accurate answers when it searches.

This is the bot that matters for AI visibility. Anthropic says disabling it prevents your content from being indexed for search, which may reduce your visibility and accuracy in user search results.

Claude-User is the third agent. When someone asks Claude a question, Claude may visit your site through Claude-User to read a page. Block it, and Claude can’t retrieve your content for that user.

ClaudeBot vs Claude-SearchBot vs Claude-User

BotWhat it doesIf you block it
ClaudeBotCollects public content that may train Anthropic’s modelsYour future content is excluded from training data
Claude-SearchBotCrawls pages to improve Claude’s search resultsPages aren’t indexed for Claude search, and visibility may drop
Claude-UserFetches a page when a Claude user asks a questionClaude can’t retrieve your page for that request

OpenAI draws the same line. GPTBot handles training and OAI-SearchBot handles search, so one policy framework covers both companies.

How to block or allow ClaudeBot in robots.txt

Anthropic’s bots follow the rules in the robots.txt file at the top level of each host. This blocks ClaudeBot across a site:

User-agent: ClaudeBot
Disallow: /

If you want out of training but still want to appear in Claude search, use a split policy. This file opts out of training and keeps search and user fetches open:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Repeat the rules on every subdomain you want covered. Anthropic tells site owners to do this for each one, so a file on www.example.com says nothing about docs.example.com.

Should you block ClaudeBot at all? It depends on what you sell. A publisher whose content is the product may opt out of training. A software company that wants Claude to describe its product well usually keeps ClaudeBot open. Training data shapes what a model already knows about a brand.

Why is ClaudeBot crawling my site so much?

If ClaudeBot is adding load to your server, you don’t have to pick between full access and a block. Anthropic supports the non-standard Crawl-delay rule, which asks the bot to wait between requests:

User-agent: ClaudeBot
Crawl-delay: 1

One trap: a bot follows the group that names it and falls back to the * group only when none does. Once you add a ClaudeBot group, ClaudeBot ignores your User-agent: * rules, so copy any Disallow lines you still need into it.

Google ignores Crawl-delay, so the same line won’t slow Googlebot. If the load continues, confirm the traffic comes from Anthropic before you change anything else.

ClaudeBot user agent and how to verify real traffic

Anthropic’s help article identifies its bots by their robots.txt tokens and doesn’t publish full user-agent strings. In your logs, filter for the tokens ClaudeBot, Claude-SearchBot and Claude-User inside the user-agent field. Match on the token, since version numbers and contact details in the string can change.

A user-agent string is plain text, though. Anyone can send a request that calls itself ClaudeBot.

Anthropic publishes the source IPs of its crawlers in a JSON list. If a request comes from an address on it, Anthropic says the crawler is coming from Anthropic. The file carries its own creation date, so use the current version for each audit.

Don’t opt out by IP block. Anthropic warns that IP blocks may not give a lasting opt-out, because they stop its bots from reading your robots.txt.

Why an allowed bot can still be blocked

Robots.txt states a preference. The request still has to pass through the rest of your stack.

robots.txt allows the bot → CDN and firewall let it through → Server returns 200 → Page is indexed for Claude search → Page can be cited

Bot-management rules on a CDN such as Cloudflare can stop AI crawlers that robots.txt allows. Anthropic also says its bots won’t try to bypass CAPTCHAs, so a challenge page ends the visit.

Look for 403 and 429 responses to Anthropic IPs in your logs. That gap between permission and real access is what AI crawler accessibility measures. Our walkthrough on checking whether AI crawlers can access your website covers each test.

Does allowing Claude-SearchBot get you into Claude answers?

It makes you eligible. Nothing more.

Anthropic says blocking the bot may reduce visibility, but it makes no promise in the other direction. Whether Claude uses a page depends on the question, the competing sources and how well the page answers it. For how search works inside the product, see Claude search.

ClaudeBot isn’t a search setting either. Anthropic documents it for training, so allowing it won’t lift you in Claude’s search results.

An example: a docs site with a split policy

Take a SaaS company with docs on docs.example.com and a blog on www.example.com. Legal wants out of model training. Marketing wants the product named when buyers ask Claude for tools.

  1. Both hosts get the split file above: ClaudeBot disallowed, Claude-SearchBot and Claude-User allowed.
  2. The CDN had a rule that challenged unknown bots. The team exempts requests from the IPs in Anthropic’s list.
  3. Server logs now show Claude-SearchBot receiving 200 responses on both hosts.
  4. From there, they track whether Claude answers cite the docs for buying questions.

The robots.txt change took ten lines. The CDN rule was the real blocker, and a robots.txt check alone would never have found it.

How to audit ClaudeBot and Claude-SearchBot access

  1. Decide your policy for each bot: training, search and user-requested fetches.
  2. Read the robots.txt file on every host and subdomain, including the * group each bot falls back to.
  3. Check CDN, firewall and bot-protection rules for challenges or blocks on Anthropic’s IPs.
  4. Filter server logs by token, confirm the IPs against Anthropic’s list and look for 403 or 429 responses.
  5. Track Claude answers for your priority prompts, so you see whether access turns into citations.

Common mistakes with Anthropic crawlers

  • Blocking ClaudeBot and assuming Claude search is blocked too. Claude-SearchBot follows its own rules.
  • Writing rules for the main domain only. Each subdomain needs its own file.
  • Trusting the bot name in logs without checking the IP list.
  • Using an IP block as the opt-out, which stops the bot from reading robots.txt.
  • Forgetting Claude-User, the agent that fetches pages for a live user question.

How Ansvisor helps

Ansvisor’s Site Audit checks robots.txt and AI bot access for any public URL. AI Traffic Analytics shows crawler activity next to AI-referred visits. Once access is fixed, the Claude AI visibility tracker shows the prompts where Claude mentions or cites your brand, next to your competitors. Ansvisor tracks Claude on Enterprise plans.

ClaudeBot, Claude Bot, Claude-SearchBot, Claude SearchBot, Anthropic Crawler, Claude-User

Let AI Search Work for Your Growth

Each prompt you win brings more mentions, more citations and more buyers. Ansvisor keeps finding the next one.

AI Answers Mentions Citations Visibility Score AI Share of Voice AI Traffic Revenue
Win Back Search VisibilityTry Ansvisor 14-day Free →✓ Zero risk. Cancel anytime.

FAQ

Frequently asked questions.

What is ClaudeBot?

ClaudeBot is Anthropic’s crawler for model development. It collects public web content that could contribute to training Claude models. Blocking it in robots.txt tells Anthropic to exclude your future content from its training datasets.

What is Claude-SearchBot?

Claude-SearchBot is Anthropic’s search crawler. It reads pages to improve the relevance and accuracy of Claude’s search results. Anthropic says blocking it stops your content from being indexed for search and may reduce your visibility in those results.

Can I block ClaudeBot but allow Claude-SearchBot?

Yes. Anthropic treats them as separate user agents, so one robots.txt file can disallow ClaudeBot and allow Claude-SearchBot. That opts you out of training while keeping your pages available to Claude’s search features.

Does ClaudeBot respect robots.txt and Crawl-delay?

Yes. Anthropic says its bots honor robots.txt and support the non-standard Crawl-delay rule. It also says they won’t try to bypass CAPTCHAs. Put the rules in the robots.txt file of every subdomain you want covered.

How do I know a ClaudeBot request is real?

Check the source IP against the list Anthropic publishes at claude.com/crawling/bots.json. A user-agent name is easy to fake, so it proves nothing on its own. Anthropic says an IP on that list means the request comes from its crawler.

Help us grow the AI Visibility Glossary

New terms are added regularly.

Help us improve the page or suggest a new term →
About the Author
Cihan Geyik

Cihan Geyik

Co-founder at Ansvisor

Cihan Geyik is the co-founder of Ansvisor, an open-source AI Visibility platform for AI Search. With more than 15 years of experience in digital marketing and growth, he writes about AI visibility, AI search, AEO, GEO, citations, and answer engines. He focuses on helping brands understand and improve their presence across ChatGPT, Gemini, Google AI Overviews, Bing AI Mode and other AI answer engines.

Google Preferred Sources
Get more Ansvisor insights in Google

Add Ansvisor as a preferred source and Google will show you more of our AI search, AEO and GEO guides.