Technical Concepts
AI Crawlers

AI Crawlers

AI crawlers are bots that fetch web pages for AI companies. They fall into three groups: training crawlers, search crawlers that feed AI answers, and fetchers that load a page when a user asks.
June 26, 2026
Cihan Geyik
Table of Content

Open a server log today and you’ll likely find several AI user agents, from GPTBot to Perplexity-User. They don’t all want the same thing. AI crawlers fetch pages for one of three jobs. They train models, build the search indexes behind AI answers, or load a page live because a user asked. Each job deserves its own robots.txt decision.

One company, several bots. OpenAI and Anthropic each run a training crawler, a search crawler and a user-triggered fetcher. Blocking one of them leaves the other two untouched.

What are the three types of AI crawlers?

TypeJobExamplesrobots.txt
Training crawlersCollect pages that may train future modelsGPTBot, ClaudeBot, CCBotHonored, per each operator
Search crawlersBuild the indexes AI answers retrieve fromOAI-SearchBot, Claude-SearchBot, PerplexityBotHonored, per each operator
User-triggered fetchersLoad a page when a user’s request needs itChatGPT-User, Claude-User, Perplexity-UserMay not apply, per OpenAI and Perplexity

Training crawlers

Training crawlers gather public pages that may be used to train future models. GPTBot does this for OpenAI and ClaudeBot for Anthropic. CCBot, run by the nonprofit Common Crawl, builds an open archive of web data that anyone can analyze.

Google and Apple handle training differently. Google-Extended and Applebot-Extended are robots.txt tokens with no crawler behind them. Googlebot and Applebot do the crawling, and the tokens decide whether that content may train models. Our Googlebot and Google-Extended page explains the split.

Search crawlers

Search crawlers build the indexes that AI answers pull sources from. OAI-SearchBot feeds ChatGPT search. Claude-SearchBot improves Claude’s search results, and PerplexityBot surfaces sites in Perplexity. Block one and that engine loses its own crawled copy of your pages.

Classic search crawlers belong in this group too. Google AI Overviews and AI Mode draw on Google’s Search index, which Googlebot builds. Microsoft Copilot draws on Bing’s index.

User-triggered fetchers

ChatGPT-User, Claude-User and Perplexity-User visit a page when someone’s request needs it, such as a pasted link or a live lookup. They don’t crawl on a schedule.

robots.txt is a weaker control here. OpenAI says its rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Anthropic’s crawler policy says blocking Claude-User stops Claude from retrieving your content for user queries. Our ClaudeBot and Claude-SearchBot page covers all three Anthropic agents.

How AI crawlers differ from search engine crawlers

Classic web crawlers like Googlebot render JavaScript and revisit pages on a steady schedule. Most AI crawlers do less.

Vercel’s study of AI crawler traffic, published in December 2024, found that the major AI crawlers didn’t render JavaScript. ChatGPT’s and Claude’s crawlers downloaded script files without running them. Gemini, which uses Googlebot’s infrastructure, and Applebot were the exceptions.

Volume is lower too, though not small. Across Vercel’s network, GPTBot made 569 million requests in a month and Claude’s crawler 370 million. Googlebot made 4.5 billion.

The lesson is plain. Put prices, specs and main text in the server-rendered HTML. Keep redirects current as well, since Vercel found about a third of ChatGPT and Claude crawler fetches hit 404 pages.

How to manage AI crawlers

List the bots in your logs → Decide per type → Write named robots.txt groups → Check CDN and firewall rules → Track citations

Most brands that want AI recommendations allow search crawlers and user-triggered fetchers, then decide on training separately. A file for that policy might look like this:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

Several user agents can share one group, and each bot follows the group that names it. A publisher that licenses its archive would keep this file. A SaaS company that wants models to know its product would likely move the training bots into the allow group.

robots.txt only states policy. CDNs and firewalls enforce access, and a one-click "block AI bots" setting can override every allow rule above. Our post on checking if AI crawlers can access your website covers those layers step by step.

Common mistakes with AI crawlers

  • Treating every AI bot as a scraper. Blocking search crawlers removes you from the answers buyers read.
  • Assuming one block per company is enough. OpenAI’s GPTBot rule does nothing to OAI-SearchBot.
  • Searching logs for a Google-Extended crawler. The token has no user agent.
  • Trusting user agent strings. Check request IPs against each operator’s published list before allowing or blocking.
  • Relying on robots.txt to protect private pages. Use a login, since user-triggered fetchers may not follow it.

How Ansvisor helps

Ansvisor’s Site Audit checks robots.txt, AI bot access and llms.txt on any page, so a stray rule doesn’t lock out a search crawler. Ansvisor then tracks whether the engines behind those crawlers mention and cite you. Coverage spans ChatGPT, Perplexity, Gemini, Copilot, Google AI Overviews and AI Mode, with Claude on Enterprise plans.

AI Bots, AI Web Crawlers, LLM Crawlers, AI Search Crawlers, AI User Agents

Let AI Search Work for Your Growth

Each prompt you win brings more mentions, more citations and more buyers. Ansvisor keeps finding the next one.

AI Answers Mentions Citations Visibility Score AI Share of Voice AI Traffic Revenue
Win Back Search VisibilityTry Ansvisor 14-day Free →✓ Zero risk. Cancel anytime.

FAQ

Frequently asked questions.

What are AI crawlers?

AI crawlers are automated agents that fetch web pages for AI companies such as OpenAI, Anthropic, Perplexity and Google. Some collect training data, some build the search indexes behind AI answers, and some load a single page when a user asks.

What are the main types of AI crawlers?

There are three. Training crawlers such as GPTBot and ClaudeBot collect content for future models. Search crawlers such as OAI-SearchBot and PerplexityBot feed AI answers. User-triggered fetchers such as ChatGPT-User visit a page when a person’s request needs it.

Which AI crawlers should I allow?

Brands that want to be cited usually allow search crawlers and user-triggered fetchers, since those feed the answers people read. Training crawlers are a separate business decision. Publishers that license content often block them, while SaaS brands often allow them.

Do AI crawlers follow robots.txt?

The automated crawlers from OpenAI, Anthropic and Perplexity follow robots.txt, according to their documentation. User-triggered fetchers are different. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it.

Do AI crawlers run JavaScript?

Most don’t. A Vercel study found the major AI crawlers did not render JavaScript, with Gemini (through Googlebot) and Applebot as exceptions. Serve core content, such as prices and product details, in the initial HTML so every crawler can read it.

Help us grow the AI Visibility Glossary

New terms are added regularly.

Help us improve the page or suggest a new term →
About the Author
Cihan Geyik

Cihan Geyik

Co-founder at Ansvisor

Cihan Geyik is the co-founder of Ansvisor, an open-source AI Visibility platform for AI Search. With more than 15 years of experience in digital marketing and growth, he writes about AI visibility, AI search, AEO, GEO, citations, and answer engines. He focuses on helping brands understand and improve their presence across ChatGPT, Gemini, Google AI Overviews, Bing AI Mode and other AI answer engines.

Google Preferred Sources
Get more Ansvisor insights in Google

Add Ansvisor as a preferred source and Google will show you more of our AI search, AEO and GEO guides.