
If you read your server logs, you’ve probably seen ClaudeBot. Anthropic runs three web agents, and each one maps to a separate decision. Should your content train Claude models? Should it appear in Claude’s search results? Should Claude fetch your page when a user asks about it? You can say yes to one and no to another.
ClaudeBot is Anthropic’s crawler for model development. It collects public web content that could contribute to training Anthropic’s generative AI models.
A block has a specific meaning. Anthropic’s crawler documentation says a ClaudeBot restriction signals that the site’s future materials should be excluded from training. Note the word future. The opt-out works going forward.
Claude-SearchBot crawls the web to improve search results for Claude users. It reads pages so Claude can return more relevant and accurate answers when it searches.
This is the bot that matters for AI visibility. Anthropic says disabling it prevents your content from being indexed for search, which may reduce your visibility and accuracy in user search results.
Claude-User is the third agent. When someone asks Claude a question, Claude may visit your site through Claude-User to read a page. Block it, and Claude can’t retrieve your content for that user.
| Bot | What it does | If you block it |
|---|---|---|
| ClaudeBot | Collects public content that may train Anthropic’s models | Your future content is excluded from training data |
| Claude-SearchBot | Crawls pages to improve Claude’s search results | Pages aren’t indexed for Claude search, and visibility may drop |
| Claude-User | Fetches a page when a Claude user asks a question | Claude can’t retrieve your page for that request |
OpenAI draws the same line. GPTBot handles training and OAI-SearchBot handles search, so one policy framework covers both companies.
Anthropic’s bots follow the rules in the robots.txt file at the top level of each host. This blocks ClaudeBot across a site:
If you want out of training but still want to appear in Claude search, use a split policy. This file opts out of training and keeps search and user fetches open:
Repeat the rules on every subdomain you want covered. Anthropic tells site owners to do this for each one, so a file on www.example.com says nothing about docs.example.com.
Should you block ClaudeBot at all? It depends on what you sell. A publisher whose content is the product may opt out of training. A software company that wants Claude to describe its product well usually keeps ClaudeBot open. Training data shapes what a model already knows about a brand.
If ClaudeBot is adding load to your server, you don’t have to pick between full access and a block. Anthropic supports the non-standard Crawl-delay rule, which asks the bot to wait between requests:
One trap: a bot follows the group that names it and falls back to the * group only when none does. Once you add a ClaudeBot group, ClaudeBot ignores your User-agent: * rules, so copy any Disallow lines you still need into it.
Google ignores Crawl-delay, so the same line won’t slow Googlebot. If the load continues, confirm the traffic comes from Anthropic before you change anything else.
Anthropic’s help article identifies its bots by their robots.txt tokens and doesn’t publish full user-agent strings. In your logs, filter for the tokens ClaudeBot, Claude-SearchBot and Claude-User inside the user-agent field. Match on the token, since version numbers and contact details in the string can change.
A user-agent string is plain text, though. Anyone can send a request that calls itself ClaudeBot.
Anthropic publishes the source IPs of its crawlers in a JSON list. If a request comes from an address on it, Anthropic says the crawler is coming from Anthropic. The file carries its own creation date, so use the current version for each audit.
Robots.txt states a preference. The request still has to pass through the rest of your stack.
Bot-management rules on a CDN such as Cloudflare can stop AI crawlers that robots.txt allows. Anthropic also says its bots won’t try to bypass CAPTCHAs, so a challenge page ends the visit.
Look for 403 and 429 responses to Anthropic IPs in your logs. That gap between permission and real access is what AI crawler accessibility measures. Our walkthrough on checking whether AI crawlers can access your website covers each test.
It makes you eligible. Nothing more.
Anthropic says blocking the bot may reduce visibility, but it makes no promise in the other direction. Whether Claude uses a page depends on the question, the competing sources and how well the page answers it. For how search works inside the product, see Claude search.
ClaudeBot isn’t a search setting either. Anthropic documents it for training, so allowing it won’t lift you in Claude’s search results.
Take a SaaS company with docs on docs.example.com and a blog on www.example.com. Legal wants out of model training. Marketing wants the product named when buyers ask Claude for tools.
The robots.txt change took ten lines. The CDN rule was the real blocker, and a robots.txt check alone would never have found it.
Ansvisor’s Site Audit checks robots.txt and AI bot access for any public URL. AI Traffic Analytics shows crawler activity next to AI-referred visits. Once access is fixed, the Claude AI visibility tracker shows the prompts where Claude mentions or cites your brand, next to your competitors. Ansvisor tracks Claude on Enterprise plans.
Each prompt you win brings more mentions, more citations and more buyers. Ansvisor keeps finding the next one.
ClaudeBot is Anthropic’s crawler for model development. It collects public web content that could contribute to training Claude models. Blocking it in robots.txt tells Anthropic to exclude your future content from its training datasets.
Claude-SearchBot is Anthropic’s search crawler. It reads pages to improve the relevance and accuracy of Claude’s search results. Anthropic says blocking it stops your content from being indexed for search and may reduce your visibility in those results.
Yes. Anthropic treats them as separate user agents, so one robots.txt file can disallow ClaudeBot and allow Claude-SearchBot. That opts you out of training while keeping your pages available to Claude’s search features.
Yes. Anthropic says its bots honor robots.txt and support the non-standard Crawl-delay rule. It also says they won’t try to bypass CAPTCHAs. Put the rules in the robots.txt file of every subdomain you want covered.
Check the source IP against the list Anthropic publishes at claude.com/crawling/bots.json. A user-agent name is easy to fake, so it proves nothing on its own. Anthropic says an IP on that list means the request comes from its crawler.
See how AI answers describe your brand, find out why competitors get picked, and close the gaps with AI agents that work alongside your team.
14-day free trial or 50% off your first month with code SAVE50.
Continue exploring key AI visibility concepts.
Measure and improve how often your brand appears in AI-generated answers.
Learn more →Strategies for increasing visibility in answer engines and AI summaries.
Learn more →Optimizing content for AI-powered discovery experiences.
Learn more →Understand how OpenAI retrieves and synthesizes information.
Learn more →AI-generated summaries that appear directly in Google Search.
Learn more →Explore how Perplexity cites and presents sources.
Learn more →References and sources used by AI systems to support answers.
Learn more →Measure the quality and influence of cited sources.
Learn more →How easily AI systems can discover and reuse your content.
Learn more →New terms are added regularly.
Help us improve the page or suggest a new term →
Co-founder at Ansvisor
Cihan Geyik is the co-founder of Ansvisor, an open-source AI Visibility platform for AI Search. With more than 15 years of experience in digital marketing and growth, he writes about AI visibility, AI search, AEO, GEO, citations, and answer engines. He focuses on helping brands understand and improve their presence across ChatGPT, Gemini, Google AI Overviews, Bing AI Mode and other AI answer engines.
Add Ansvisor as a preferred source and Google will show you more of our AI search, AEO and GEO guides.
© 2026 Ansvisor. All rights reserved. Ansvisor is an open-source AI Search Execution Platform for AI Visibility. Ansvisor is a brand owned and operated by Empler AI Inc.
The open-source AI Search Execution Platform. Track how AI answers talk about your brand and act on it.
Open source · MIT license© 2026 Ansvisor, a brand of Empler AI Inc. · 1401 PE Ave. Unit 105, Wilmington, DE 19806, United States
Cookie preferences
We use cookies to keep the site working and, with your OK, to measure traffic and improve our marketing. Read our Cookies & Tracking policy.