
Every crawler that follows the rules checks one file before it fetches your pages: robots.txt. For years it mattered mostly for Googlebot and Bingbot. Today the same file decides whether AI platforms can read your site for search, and it signals whether your content may be used for model training. One misplaced line can switch off a whole channel.
Robots.txt is a plain text file at the top level of a host, for example https://www.example.com/robots.txt. It holds groups of rules. Each group names one or more crawlers and lists the paths they may or may not fetch.
A file covers one host, protocol and port. Rules on www.example.com don’t apply to docs.example.com, so every subdomain needs its own file.
Google describes the file’s main job as managing crawler traffic so your server isn’t overloaded. Its robots.txt introduction also states that the file isn’t a way to keep a page out of Google.
Matching follows a few rules from the standard:
| Field | What it does | Example |
|---|---|---|
| User-agent | Names the crawler the group applies to | User-agent: Googlebot |
| Disallow | A path the crawler should not fetch | Disallow: /cart/ |
| Allow | A path it may fetch, such as an exception inside a blocked folder | Allow: /cart/help |
| Sitemap | The absolute URL of a sitemap | Sitemap: https://www.example.com/sitemap.xml |
Allow is how you carve out exceptions. This group blocks a folder but keeps one page open:
The Sitemap line isn’t tied to a group, so every crawler can use it to find your XML sitemap. Google ignores fields it doesn’t support, including Crawl-delay.
A modern starting file for a marketing site might look like this:
Every crawler can reach the public pages, and OpenAI’s training crawler is opted out. OAI-SearchBot falls under the * group, so ChatGPT search keeps access. Blocking training bots is a business call. Brands that want models to know their product may prefer to leave them open.
The Robots Exclusion Protocol, or REP, is the standard that robots.txt implements. Martijn Koster defined it in 1994. It worked as an informal convention for decades, until the IETF published it as RFC 9309 in September 2022.
The RFC settled questions that crawlers used to answer differently:
The error rules can catch teams out. A robots.txt URL that keeps returning server errors can pause crawling, while a 404 removes every restriction.
AI companies publish separate tokens for separate jobs. Treat each one as its own decision.
| Token | Owner | Purpose |
|---|---|---|
| GPTBot | OpenAI | Content that may train foundation models |
| OAI-SearchBot | OpenAI | ChatGPT search |
| ClaudeBot | Anthropic | Content that may train Claude models |
| Claude-SearchBot | Anthropic | Claude search results |
| Google-Extended | Gemini training and grounding, with no effect on Google Search |
Google-Extended is a control token only. Crawling still runs on Google’s usual user agents. AI Overviews draw on pages that are indexed and eligible for a snippet, so Googlebot rules govern them.
For bot-by-bot detail, see OAI-SearchBot, ClaudeBot and Claude-SearchBot and Googlebot and Google-Extended. Robots.txt only grants or denies access. Guidance about which pages matter belongs in a separate proposal, llms.txt.
Robots.txt controls crawling. A noindex rule controls indexing. They solve different problems.
Google says a page blocked in robots.txt can still be indexed if other sites link to it, and the result then shows without a description. A crawler that can’t fetch the page also never sees its noindex tag.
Disallow: / left over from a staging site.Start with the robots.txt report in Google Search Console. It shows the files Google found for your top hosts, when each was last crawled and any parse errors. You can request a recrawl after a fix.
Robots.txt is also only one layer. A CDN rule or firewall can still block a bot the file allows, and AI crawlers have no equivalent console. Check server logs for each bot’s token and response codes, and recheck the file after every deploy. Expect up to a day before crawlers notice a change, since they may cache the file for 24 hours.
Verify tokens at the source, too. In Ansvisor tracking data, September 15 to October 14, 2026, a GitHub list that includes AI crawler user agents drew 665 citations in AI answers. OpenAI’s own crawler page drew 239. Lists go stale, so confirm each token in the vendor’s docs before you ship a rule.
Our walkthrough on checking whether AI crawlers can access your website covers the log checks step by step.
Ansvisor’s Site Audit checks robots.txt, AI bot access and llms.txt for any public URL, as part of 47 weighted AEO and GEO signals. Once bots can reach your pages, Citations Monitoring shows which of your URLs AI answers cite and which competitor pages they cite instead.
Each prompt you win brings more mentions, more citations and more buyers. Ansvisor keeps finding the next one.
Robots.txt is a text file at the root of a site, such as example.com/robots.txt. It lists rules that tell crawlers which paths they may fetch. Compliant crawlers read it before crawling, but it isn’t a security control.
The Robots Exclusion Protocol is the standard behind robots.txt. Martijn Koster defined it in 1994, and the IETF published it as RFC 9309 in September 2022. It sets the file location, the rule syntax and how crawlers handle errors.
Create a plain text file named robots.txt, add a User-agent line followed by Allow or Disallow rules, and add your sitemap URL. Upload it to the root of each host, then check Search Console’s robots.txt report for errors.
Add a group for each bot’s token, such as GPTBot or ClaudeBot, with Disallow: /. Check each vendor’s docs first, since search and training bots are usually separate. Blocking a search bot like OAI-SearchBot removes you from that platform’s search answers.
No. Robots.txt controls crawling, and noindex controls indexing. Google says a page blocked in robots.txt can still be indexed if other sites link to it. To keep a page out of results, allow crawling and use noindex.
See how AI answers describe your brand, find out why competitors get picked, and close the gaps with AI agents that work alongside your team.
14-day free trial or 50% off your first month with code SAVE50.
Continue exploring key AI visibility concepts.
Measure and improve how often your brand appears in AI-generated answers.
Learn more →Strategies for increasing visibility in answer engines and AI summaries.
Learn more →Optimizing content for AI-powered discovery experiences.
Learn more →Understand how OpenAI retrieves and synthesizes information.
Learn more →AI-generated summaries that appear directly in Google Search.
Learn more →Explore how Perplexity cites and presents sources.
Learn more →References and sources used by AI systems to support answers.
Learn more →Measure the quality and influence of cited sources.
Learn more →How easily AI systems can discover and reuse your content.
Learn more →New terms are added regularly.
Help us improve the page or suggest a new term →
Co-founder at Ansvisor
Cihan Geyik is the co-founder of Ansvisor, an open-source AI Visibility platform for AI Search. With more than 15 years of experience in digital marketing and growth, he writes about AI visibility, AI search, AEO, GEO, citations, and answer engines. He focuses on helping brands understand and improve their presence across ChatGPT, Gemini, Google AI Overviews, Bing AI Mode and other AI answer engines.
Add Ansvisor as a preferred source and Google will show you more of our AI search, AEO and GEO guides.
© 2026 Ansvisor. All rights reserved. Ansvisor is an open-source AI Search Execution Platform for AI Visibility. Ansvisor is a brand owned and operated by Empler AI Inc.
The open-source AI Search Execution Platform. Track how AI answers talk about your brand and act on it.
Open source · MIT license© 2026 Ansvisor, a brand of Empler AI Inc. · 1401 PE Ave. Unit 105, Wilmington, DE 19806, United States
Cookie preferences
We use cookies to keep the site working and, with your OK, to measure traffic and improve our marketing. Read our Cookies & Tracking policy.