



OpenAI runs three bots, and each one does a different job. Anthropic runs three more. Google and Apple handle AI training through separate control tokens. So a single "allow AI" or "block AI" rule rarely does what you expect.
You need to know which name does what before you touch robots.txt. We cover each user agent first, then seven checks you can run yourself. Every robots.txt example was checked against the vendor’s documentation in October 2026.
An AI crawler is an automated user agent that fetches web pages for an AI company. The pages may train a model, feed an AI search index, or answer a question a user just asked.
Each crawler identifies itself with a name, such as GPTBot or ClaudeBot. Your robots.txt file uses those names to allow or block each one.
AI bots fall into three groups. The group tells you what you gain or lose when you block a bot.
Blocking a training crawler won’t remove you from AI search answers. Blocking a search crawler or a user fetcher can.
Google-Extended and Applebot-Extended are a different case. Neither has a crawler of its own. They only tell Google and Apple how pages crawled by Googlebot or Applebot may be used.
Most brands should allow search crawlers and user fetchers. Whether to allow training is a business and legal call, and it has nothing to do with visibility.
These user agents matter for ChatGPT, Claude, Google AI Overviews, AI Mode, Gemini, Microsoft Copilot, Perplexity and Apple. Each behavior below comes from the vendor’s own documentation.
OpenAI’s documentation on its crawlers says robots.txt changes can take about 24 hours to reach search results. OpenAI also publishes IP address lists for each bot.
Anthropic’s help article on its bots says they honor robots.txt and support the non-standard Crawl-delay rule. It advises against blocking IP addresses, because that stops the bots from reading robots.txt.
Anthropic also asks you to add rules on every subdomain, since each subdomain has its own robots.txt file.
Google’s list of common crawlers says Google-Extended does not affect inclusion or ranking in Google Search. Blocking it can still limit how Gemini Apps use your pages for grounding.
To leave AI Overviews and AI Mode without leaving Search, use the Search generative AI control in Search Console settings. Google’s Search Console Help says it rolled out to all sites on August 31, 2026.
Bingbot is Bing’s standard crawler, described in Bing’s crawler documentation. Microsoft reports Copilot citations in Bing Webmaster Tools, so Bingbot access is the first check for Copilot.
For finer control, Microsoft announced NOCACHE and NOARCHIVE robots meta values in 2023 for Bing Chat, now Microsoft Copilot. It said NOARCHIVE pages are left out of those answers and out of model training.
Perplexity’s crawler documentation says changes may take up to 24 hours to apply. It also publishes IP lists for both agents.
One detail catches teams out. Apple’s Applebot page says that if robots.txt names Googlebot but not Applebot, Applebot follows the Googlebot rules.
Open https://www.yoursite.com/robots.txt in a browser. Repeat this for every subdomain that hosts content, such as a blog or docs subdomain.
For each bot in the list above, find the group it obeys. A bot follows the group that names it, and only falls back to User-agent: * when no group names it.
This rule comes from the robots.txt standard, RFC 9309, and it causes a lot of accidental gaps. Add a group for GPTBot, and GPTBot stops reading the Disallow lines in your User-agent: * group.
Robots.txt controls crawling, but page tags control what can be shown. Check the HTML head and the HTTP headers of the pages that matter most.
Request an important page with an AI user agent and read the status code. You can run this from any terminal:
curl -I -A "GPTBot" https://www.yoursite.com/your-page
A 200 status means the server returned the page. A 403, 429, 503 or a challenge page means something between the bot and your content is refusing it.
This test copies the user-agent name only, not the bot’s IP address. So it catches rules that match the name and misses rules that check IP ranges.
Plenty of blocks happen outside robots.txt. CDNs, web application firewalls and bot managers can challenge or reject AI crawlers that robots.txt allows.
Some CDNs, such as Cloudflare, offer one-click settings to block AI crawlers. Look for those toggles in your security dashboard, along with rate limits and any country or IP rules.
Don’t open your firewall to anything that claims to be a known bot. User-agent strings are easy to fake. Allow bots by verified IP lists or through your provider’s verified-bot feature.
Not every AI crawler is documented to run JavaScript. If your main copy or pricing loads only after scripts run, a crawler may get an empty page.
View the page source, not the rendered page, and search for a sentence from your main content. If it’s missing, server-side rendering or static HTML fixes the problem.
Configuration shows what should happen. Logs show what does happen.
If an AI crawler never shows up in your logs, check robots.txt and firewall rules first, then internal links. A page that nothing links to is hard for any crawler to find.
Google Search Console has a robots.txt report and a URL Inspection tool that shows whether Googlebot can fetch and index a page. Bing Webmaster Tools has a robots.txt tester and URL inspection.
Both now report AI visibility as well. Search Console shows impressions in AI Overviews and AI Mode. Bing Webmaster Tools shows citations in Microsoft Copilot.
Treat these templates as a starting point. Each line below is one directive, and each User-agent line starts a new group in your file.
This policy opts out of model training wherever a vendor offers a separate control. AI search crawlers still get in, and so do the classic Google and Bing crawlers.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Disallow: /admin/
OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and Applebot have no group of their own here. They follow the User-agent: * group and can crawl everything except /admin/.
Mind the Gemini trade-off. Blocking Google-Extended also limits grounding in Gemini Apps. If Gemini visibility matters more to you, skip that group.
If you want maximum reach in AI answers and accept training use, one group is enough.
User-agent: *
Allow: /
When you add a named group, copy every Disallow line from the User-agent: * group into it. Skip that step and the named bot can crawl paths you meant to keep private.
User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/
Disallow: /checkout/
Worried about crawl volume? Anthropic also supports a Crawl-delay line, such as Crawl-delay: 1 under User-agent: ClaudeBot.
Allowing a crawler makes your pages eligible, and that’s all it does. No AI answer is obliged to mention or cite you.
Once access is fixed, track whether your pages get cited for the prompts that matter. Our AI visibility audit checklist covers the content and citation checks that come next, including brand signals.
If your pages are crawlable but still missing, read why a brand doesn’t appear in ChatGPT or Google AI answers.
Ansvisor, an open-source AI search intelligence platform, puts AI crawler activity and AI referral visits side by side in AI Traffic Analytics.
The AI Visibility Site Audit checks robots.txt, AI bot access and canonical URLs as part of 47 weighted AEO and GEO signals for any public URL.
Once bots can reach your pages, Ansvisor tracks whether ChatGPT, Gemini, AI Overviews, AI Mode and Copilot cite them. For the referral side, read how to track traffic from ChatGPT and AI search.
Start a 14-day free trial to check your crawler access and AI visibility, or self-host the open-source project on GitHub.
Open yoursite.com/robots.txt and note which group each AI user agent follows. Then request an important page with that user agent and check for a 200 response. Finally, look for verified bot visits in your server or CDN logs.
No. OpenAI uses GPTBot for model training and OAI-SearchBot for ChatGPT search. Blocking GPTBot opts you out of training. Blocking OAI-SearchBot is what keeps your pages out of ChatGPT search answers.
No. Google says Google-Extended does not affect inclusion in Google Search, which includes AI Overviews and AI Mode. It controls Gemini training and grounding in Gemini Apps and Vertex AI. To leave AI Overviews and AI Mode, Google offers a separate setting in Search Console.
Check your server or CDN logs for AI user agents such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. Confirm the IP addresses against vendor lists, since user agents can be faked. If you see no visits for weeks, check robots.txt and firewall rules first, then internal links.
The message usually means a rule blocks automated access to that URL. Check robots.txt for a Disallow line that matches the bot and path. Then look for noindex tags, and for firewall rules that return 403 errors.
Co-founder at Ansvisor
Cihan Geyik is the co-founder of Ansvisor, an open-source, cloud-ready AI Visibility platform for AI Search. With more than 15 years of experience in digital marketing and growth, he writes about AI visibility, AI search, AEO, GEO, citations, and answer engines. He focuses on helping brands understand and improve their presence across ChatGPT, Gemini, Perplexity, Google AI Overviews, and other AI-powered discovery platforms.
Each prompt you win brings more mentions, more citations and more buyers. Ansvisor keeps finding the next one.
Add Ansvisor as a preferred source and Google will show you more of our AI search, AEO and GEO guides.
© 2026 Ansvisor. All rights reserved. Ansvisor is an open-source AI Search Execution Platform for AI Visibility. Ansvisor is a brand owned and operated by Empler AI Inc.
The open-source AI Search Execution Platform. Track how AI answers talk about your brand and act on it.
Open source · MIT license© 2026 Ansvisor, a brand of Empler AI Inc. · 1401 PE Ave. Unit 105, Wilmington, DE 19806, United States
Cookie preferences
We use cookies to keep the site working and, with your OK, to measure traffic and improve our marketing. Read our Cookies & Tracking policy.

