ClaudeBot / Claude-SearchBot (Anthropic Crawlers)

ClaudeBot and Claude-SearchBot are web crawlers operated by Anthropic, but they serve different purposes. ClaudeBot collects public web content that could potentially contribute to the training of Anthropic's generative AI models, while Claude-SearchBot navigates and analyzes the web to improve the relevance and accuracy of search results for Claude users.

Anthropic also operates a third web agent called Claude-User, which can retrieve website content when a Claude user initiates a request.

Keeping these three identities separate is important for AI Crawler Accessibility because website owners can make different decisions about model training, search visibility, and user-directed retrieval.

What Is ClaudeBot?

ClaudeBot is Anthropic's model-development crawler. Anthropic states that ClaudeBot helps improve the utility and safety of its generative AI models by collecting web content that could potentially contribute to their training.

This makes ClaudeBot fundamentally different from Claude-SearchBot. ClaudeBot's documented purpose concerns potential model-training data, rather than improving Claude's web search results.

What Is Claude-SearchBot?

Claude-SearchBot is Anthropic's web crawler for search. Anthropic states that it navigates the web to improve search result quality for users and analyzes online content to improve the relevance and accuracy of search responses.

Anthropic further explains that disabling Claude-SearchBot prevents its systems from indexing the site's content for search optimization. This may reduce the site's visibility and accuracy in user search results.

ClaudeBot vs. Claude-SearchBot

ClaudeBot and Claude-SearchBot belong to the same Anthropic crawler ecosystem, but they should not be treated as interchangeable.

How to Allow ClaudeBot

A website that wants to explicitly permit ClaudeBot to crawl its public content can use a crawler-specific robots.txt group:

These instructions belong in the site's robots.txt file.

How to Block ClaudeBot

A website that does not want ClaudeBot crawling its content can use:

Anthropic states that restricting ClaudeBot signals that future materials from the site should be excluded from its AI model training datasets.

This rule concerns ClaudeBot specifically. It does not automatically mean that Claude-SearchBot or Claude-User has been blocked.

How to Allow Claude-SearchBot

A site that wants to explicitly allow Anthropic's search crawler can use:

This communicates that Claude-SearchBot is permitted to crawl the public site under the applicable Robots Exclusion Protocol rules.

How to Block Claude-SearchBot

A website can block Claude-SearchBot with:

Anthropic states that disabling Claude-SearchBot prevents its system from indexing the site's content for search optimization and may reduce the site's visibility and accuracy in user search results.

Can You Block ClaudeBot but Allow Claude-SearchBot?

Yes. Because Anthropic identifies these bots separately, website owners can define separate robots.txt policies for them.

For example, a publisher that wants its content accessible for Claude search but does not want ClaudeBot crawling future content for potential training use could configure:

This distinction is particularly important for organizations that want to separate AI Search visibility from AI model-training permissions.

What Is Claude-User?

Claude-User is Anthropic's user-directed web retrieval agent.

When someone asks Claude a question, Claude may access a website using the Claude-User agent to retrieve content in response to that request.

This differs from ClaudeBot's model-development crawling and Claude-SearchBot's search-quality crawling.

Does Claude-User Respect Robots.txt?

Anthropic states that its bots honor industry-standard do not crawl directives in robots.txt. Its crawler documentation includes ClaudeBot, Claude-SearchBot, and Claude-User within this bot framework.

Anthropic also states that disabling Claude-User prevents its system from retrieving website content in response to a user query, which may reduce the site's visibility for user-directed web search.

This makes Claude-User an important additional consideration when evaluating Claude accessibility rather than focusing exclusively on Claude-SearchBot.

How to Control All Three Anthropic Bots Separately

A publisher can define a different policy for each Anthropic bot. For example:

This configuration communicates three separate preferences:

• allow Anthropic's search crawler;
• allow user-directed retrieval through Claude-User; and
• restrict ClaudeBot's model-training-related crawling.

The appropriate configuration depends on the website owner's own content, search, and AI training policies.

Do Anthropic Crawlers Respect Robots.txt?

Yes. Anthropic states that its bots respect "do not crawl" signals by honoring industry-standard robots.txt directives.

Anthropic also says its crawlers are designed to minimize disruption and that it supports the non-standard Crawl-delay extension to robots.txt.

Crawl-delay can be useful when the goal is to reduce crawl frequency rather than block access entirely.

Anthropic Crawlers and Subdomains

Anthropic instructs website owners to apply their robots.txt preferences to every subdomain they want to opt out.

For example, policies for a main website may not automatically represent the intended policy for separate hosts such as:

• www.example.com;
• docs.example.com;
• blog.example.com; and
• app.example.com.

Teams auditing Crawler Access should therefore review all relevant hosts rather than checking only the primary domain.

Claude Crawlers and Crawl-delay

Anthropic explicitly supports the non-standard Crawl-delay robots.txt extension as a way to limit crawling activity.

This gives publishers an option between unrestricted crawling and a complete block when server load or crawl frequency is the primary concern.

Do Anthropic Crawlers Bypass CAPTCHAs?

Anthropic states that its bots respect anti-circumvention technologies and that it will not attempt to bypass CAPTCHAs for the sites it crawls.

This also illustrates why AI Crawler Accessibility involves more than robots.txt.

A crawler may be allowed by robots.txt while technical infrastructure prevents successful retrieval.

Robots.txt Permission vs. Actual Claude Access

Allowing ClaudeBot or Claude-SearchBot in robots.txt does not guarantee that the crawler can successfully retrieve the site.

Other technical layers can include:

• CDNs;
• web application firewalls;
• bot-management systems;
• CAPTCHA challenges;
• authentication;
• IP restrictions;
• geographic restrictions;
• rate limits;
• HTTP 403 responses;
• HTTP 429 responses; and
• server errors.

The difference between crawler permission and successful retrieval is a core part of Crawler Access.

How to Verify Anthropic Crawler Traffic

User-agent strings alone can be imitated, so identifying a bot name in server logs does not necessarily prove that the request originated from Anthropic.

Anthropic publishes a maintained list of source IP ranges for its crawlers. Anthropic states that a crawler request originating from an IP address on this list indicates that the crawler is coming from Anthropic.

The current list is available from Anthropic's crawler IP list.

Because infrastructure can change, teams should reference Anthropic's maintained list rather than permanently copying individual addresses into documentation.

Should You Block Anthropic Crawlers by IP Address?

Anthropic recommends using robots.txt to communicate crawler opt-out preferences.

Anthropic specifically warns that alternative approaches such as blocking crawler IP addresses may not work correctly or persistently guarantee an opt-out because IP blocking can prevent its bots from reading the site's robots.txt file.

Claude-SearchBot and AI Search Indexing

Claude-SearchBot is directly relevant to AI Search Indexing.

Anthropic states that disabling Claude-SearchBot prevents its system from indexing a site's content for search optimization.

However, indexing should not be treated as equivalent to appearing in every Claude answer.

Each stage is distinct, and successful crawler access alone does not guarantee downstream visibility.

Does Allowing Claude-SearchBot Guarantee Claude Visibility?

No. Allowing Claude-SearchBot removes one potential technical barrier to Anthropic's search crawler, but it does not guarantee that a particular page or brand will appear for a specific query.

Anthropic states that blocking Claude-SearchBot may reduce a site's visibility in user search results. That should not be reversed into a claim that allowing the crawler guarantees visibility.

Does Allowing ClaudeBot Improve Claude Search Visibility?

ClaudeBot should not be treated as a Claude Search ranking mechanism.

Anthropic documents ClaudeBot for collecting content that could potentially contribute to model training, while Claude-SearchBot is the crawler specifically associated with search result quality.

Claude Crawlers and AI Visibility

For teams working on AI Visibility, the Anthropic crawler architecture creates an important distinction between technical eligibility and measured performance.

Claude-SearchBot accessibility can support search discovery, while Claude-User accessibility can affect user-directed retrieval. Neither automatically proves that a brand is appearing in relevant Claude answers.

Claude-SearchBot and AI Citations

Search crawler accessibility can be part of the technical foundation for source visibility, but it should not be used as a substitute for measuring actual AI Citations.

A page can be technically accessible without being selected as a source for a particular query.

This is why crawler configuration and citation monitoring should be evaluated as separate layers of an AI Search strategy.

How to Audit ClaudeBot and Claude-SearchBot

1. Identify your policy. Decide separately whether you want to permit model-training crawling, search crawling, and user-directed retrieval.
2. Review robots.txt. Check explicit rules for ClaudeBot, Claude-SearchBot, and Claude-User.
3. Review every relevant subdomain. Do not assume one robots.txt file represents the policy for all hosts.
4. Check important URL paths. Verify that public content intended for Claude search is not unintentionally blocked.
5. Inspect infrastructure. Check CDN, WAF, bot protection, CAPTCHA, rate limits, and HTTP responses.
6. Review crawl frequency. Consider Anthropic's Crawl-delay support when crawl load is the problem rather than access itself.
7. Verify crawler traffic. Compare suspicious crawler traffic with Anthropic's current published crawler IP information.
8. Measure downstream visibility. Do not assume successful crawling means successful Claude visibility.

Common Claude Crawler Mistakes

• treating ClaudeBot and Claude-SearchBot as the same crawler;
• blocking ClaudeBot and assuming Claude search is also blocked;
• allowing ClaudeBot and assuming it improves Claude search rankings;
• forgetting Claude-User when auditing user-directed retrieval;
• allowing a bot in robots.txt while blocking it at the infrastructure layer;
• checking only the main domain while ignoring subdomains;
• using IP blocking as a replacement for documented robots.txt preferences;
• assuming crawler access guarantees AI citations;
• assuming indexing guarantees appearance in an answer; and
• treating all Anthropic web traffic as model-training traffic.

Claude Crawlers and LLM SEO

Understanding Anthropic's crawler architecture is one technical component of LLM SEO, AEO, and Generative Engine Optimization (GEO).

The objective is not simply to allow every AI crawler. It is to define deliberate policies for different crawler purposes and ensure that search-related systems are not unintentionally prevented from accessing content intended for public discovery.

From Claude Crawler Access to Claude Visibility

Crawler access is an input. Actual visibility is the outcome that needs to be measured.

Ansvisor's Claude AI Visibility Tracker helps teams monitor how brands appear across relevant Claude prompts and answers.

Ansvisor's Citation Monitoring can then help measure which domains and URLs actually appear as sources instead of inferring citation performance from crawler configuration.

The broader Ansvisor AI Search Intelligence Platform connects AI visibility, prompts, citations, competitors, opportunities, and actions across the AI Search workflow.