
Open a server log today and you’ll likely find several AI user agents, from GPTBot to Perplexity-User. They don’t all want the same thing. AI crawlers fetch pages for one of three jobs. They train models, build the search indexes behind AI answers, or load a page live because a user asked. Each job deserves its own robots.txt decision.
| Type | Job | Examples | robots.txt |
|---|---|---|---|
| Training crawlers | Collect pages that may train future models | GPTBot, ClaudeBot, CCBot | Honored, per each operator |
| Search crawlers | Build the indexes AI answers retrieve from | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Honored, per each operator |
| User-triggered fetchers | Load a page when a user’s request needs it | ChatGPT-User, Claude-User, Perplexity-User | May not apply, per OpenAI and Perplexity |
Training crawlers gather public pages that may be used to train future models. GPTBot does this for OpenAI and ClaudeBot for Anthropic. CCBot, run by the nonprofit Common Crawl, builds an open archive of web data that anyone can analyze.
Google and Apple handle training differently. Google-Extended and Applebot-Extended are robots.txt tokens with no crawler behind them. Googlebot and Applebot do the crawling, and the tokens decide whether that content may train models. Our Googlebot and Google-Extended page explains the split.
Search crawlers build the indexes that AI answers pull sources from. OAI-SearchBot feeds ChatGPT search. Claude-SearchBot improves Claude’s search results, and PerplexityBot surfaces sites in Perplexity. Block one and that engine loses its own crawled copy of your pages.
Classic search crawlers belong in this group too. Google AI Overviews and AI Mode draw on Google’s Search index, which Googlebot builds. Microsoft Copilot draws on Bing’s index.
ChatGPT-User, Claude-User and Perplexity-User visit a page when someone’s request needs it, such as a pasted link or a live lookup. They don’t crawl on a schedule.
robots.txt is a weaker control here. OpenAI says its rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Anthropic’s crawler policy says blocking Claude-User stops Claude from retrieving your content for user queries. Our ClaudeBot and Claude-SearchBot page covers all three Anthropic agents.
Classic web crawlers like Googlebot render JavaScript and revisit pages on a steady schedule. Most AI crawlers do less.
Vercel’s study of AI crawler traffic, published in December 2024, found that the major AI crawlers didn’t render JavaScript. ChatGPT’s and Claude’s crawlers downloaded script files without running them. Gemini, which uses Googlebot’s infrastructure, and Applebot were the exceptions.
Volume is lower too, though not small. Across Vercel’s network, GPTBot made 569 million requests in a month and Claude’s crawler 370 million. Googlebot made 4.5 billion.
The lesson is plain. Put prices, specs and main text in the server-rendered HTML. Keep redirects current as well, since Vercel found about a third of ChatGPT and Claude crawler fetches hit 404 pages.
Most brands that want AI recommendations allow search crawlers and user-triggered fetchers, then decide on training separately. A file for that policy might look like this:
Several user agents can share one group, and each bot follows the group that names it. A publisher that licenses its archive would keep this file. A SaaS company that wants models to know its product would likely move the training bots into the allow group.
robots.txt only states policy. CDNs and firewalls enforce access, and a one-click "block AI bots" setting can override every allow rule above. Our post on checking if AI crawlers can access your website covers those layers step by step.
Ansvisor’s Site Audit checks robots.txt, AI bot access and llms.txt on any page, so a stray rule doesn’t lock out a search crawler. Ansvisor then tracks whether the engines behind those crawlers mention and cite you. Coverage spans ChatGPT, Perplexity, Gemini, Copilot, Google AI Overviews and AI Mode, with Claude on Enterprise plans.
Each prompt you win brings more mentions, more citations and more buyers. Ansvisor keeps finding the next one.
AI crawlers are automated agents that fetch web pages for AI companies such as OpenAI, Anthropic, Perplexity and Google. Some collect training data, some build the search indexes behind AI answers, and some load a single page when a user asks.
There are three. Training crawlers such as GPTBot and ClaudeBot collect content for future models. Search crawlers such as OAI-SearchBot and PerplexityBot feed AI answers. User-triggered fetchers such as ChatGPT-User visit a page when a person’s request needs it.
Brands that want to be cited usually allow search crawlers and user-triggered fetchers, since those feed the answers people read. Training crawlers are a separate business decision. Publishers that license content often block them, while SaaS brands often allow them.
The automated crawlers from OpenAI, Anthropic and Perplexity follow robots.txt, according to their documentation. User-triggered fetchers are different. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it.
Most don’t. A Vercel study found the major AI crawlers did not render JavaScript, with Gemini (through Googlebot) and Applebot as exceptions. Serve core content, such as prices and product details, in the initial HTML so every crawler can read it.
See how AI answers describe your brand, find out why competitors get picked, and close the gaps with AI agents that work alongside your team.
14-day free trial or 50% off your first month with code SAVE50.
Continue exploring key AI visibility concepts.
Measure and improve how often your brand appears in AI-generated answers.
Learn more →Strategies for increasing visibility in answer engines and AI summaries.
Learn more →Optimizing content for AI-powered discovery experiences.
Learn more →Understand how OpenAI retrieves and synthesizes information.
Learn more →AI-generated summaries that appear directly in Google Search.
Learn more →Explore how Perplexity cites and presents sources.
Learn more →References and sources used by AI systems to support answers.
Learn more →Measure the quality and influence of cited sources.
Learn more →How easily AI systems can discover and reuse your content.
Learn more →New terms are added regularly.
Help us improve the page or suggest a new term →
Co-founder at Ansvisor
Cihan Geyik is the co-founder of Ansvisor, an open-source AI Visibility platform for AI Search. With more than 15 years of experience in digital marketing and growth, he writes about AI visibility, AI search, AEO, GEO, citations, and answer engines. He focuses on helping brands understand and improve their presence across ChatGPT, Gemini, Google AI Overviews, Bing AI Mode and other AI answer engines.
Add Ansvisor as a preferred source and Google will show you more of our AI search, AEO and GEO guides.
© 2026 Ansvisor. All rights reserved. Ansvisor is an open-source AI Search Execution Platform for AI Visibility. Ansvisor is a brand owned and operated by Empler AI Inc.
The open-source AI Search Execution Platform. Track how AI answers talk about your brand and act on it.
Open source · MIT license© 2026 Ansvisor, a brand of Empler AI Inc. · 1401 PE Ave. Unit 105, Wilmington, DE 19806, United States
Cookie preferences
We use cookies to keep the site working and, with your OK, to measure traffic and improve our marketing. Read our Cookies & Tracking policy.