AI & Infrastructure
Robots.txt

Robots.txt: Rules for Search and AI Crawlers

Robots.txt is a plain text file at the root of a website that tells crawlers which URLs they may or may not fetch. It follows the Robots Exclusion Protocol, standardized as RFC 9309.
October 5, 2026
Cihan Geyik
Table of Content

Every crawler that follows the rules checks one file before it fetches your pages: robots.txt. For years it mattered mostly for Googlebot and Bingbot. Today the same file decides whether AI platforms can read your site for search, and it signals whether your content may be used for model training. One misplaced line can switch off a whole channel.

Robots.txt is a request, not a lock. Compliant crawlers follow it, but RFC 9309 says the rules "are not a form of access authorization." Protect private content with authentication.

What is robots.txt?

Robots.txt is a plain text file at the top level of a host, for example https://www.example.com/robots.txt. It holds groups of rules. Each group names one or more crawlers and lists the paths they may or may not fetch.

A file covers one host, protocol and port. Rules on www.example.com don’t apply to docs.example.com, so every subdomain needs its own file.

Google describes the file’s main job as managing crawler traffic so your server isn’t overloaded. Its robots.txt introduction also states that the file isn’t a way to keep a page out of Google.

How does robots.txt work?

Crawler requests /robots.txt → Finds the group that names it → Falls back to the * group if none does → Applies the most specific matching rule → Fetches or skips the URL

Matching follows a few rules from the standard:

  • User-agent names match regardless of case. Paths are case-sensitive, so /Blog/ and /blog/ are different.
  • If several groups name the same crawler, their rules are combined into one.
  • When Allow and Disallow both match a URL, the longer rule wins. On a tie, Allow should win.
  • Crawlers must support two special characters: * for any sequence of characters and $ for the end of the URL.

Robots.txt syntax: Allow, Disallow and Sitemap

FieldWhat it doesExample
User-agentNames the crawler the group applies toUser-agent: Googlebot
DisallowA path the crawler should not fetchDisallow: /cart/
AllowA path it may fetch, such as an exception inside a blocked folderAllow: /cart/help
SitemapThe absolute URL of a sitemapSitemap: https://www.example.com/sitemap.xml

Allow is how you carve out exceptions. This group blocks a folder but keeps one page open:

User-agent: *
Disallow: /internal/
Allow: /internal/press-kit.html

The Sitemap line isn’t tied to a group, so every crawler can use it to find your XML sitemap. Google ignores fields it doesn’t support, including Crawl-delay.

How to create a robots.txt file

  1. Create a plain UTF-8 text file named robots.txt, all lowercase.
  2. Start with a group for all crawlers, then add groups for bots that need different rules.
  3. Add your sitemap URL at the end.
  4. Upload it to the root of each host. Most CMSs and SEO plugins let you edit it without FTP.
  5. Open the live URL in a browser, then check Search Console’s robots.txt report for errors.

A modern starting file for a marketing site might look like this:

User-agent: *
Disallow: /account/
Disallow: /cart/

User-agent: GPTBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Every crawler can reach the public pages, and OpenAI’s training crawler is opted out. OAI-SearchBot falls under the * group, so ChatGPT search keeps access. Blocking training bots is a business call. Brands that want models to know their product may prefer to leave them open.

What is the Robots Exclusion Protocol?

The Robots Exclusion Protocol, or REP, is the standard that robots.txt implements. Martijn Koster defined it in 1994. It worked as an informal convention for decades, until the IETF published it as RFC 9309 in September 2022.

The RFC settled questions that crawlers used to answer differently:

  • The file must be named /robots.txt, all lowercase, at the top level of the service.
  • Crawlers shouldn’t use a cached copy for more than 24 hours, unless the file can’t be reached.
  • Crawlers may cap how much of the file they parse, but the cap must be at least 500 KiB. Google stops at 500 KiB.
  • A 4xx response means there’s no file, so crawlers may fetch anything. A 5xx response means they must assume a full disallow.
  • Crawlers should follow at least five redirects to reach the file.

The error rules can catch teams out. A robots.txt URL that keeps returning server errors can pause crawling, while a 404 removes every restriction.

Robots.txt and AI crawlers

AI companies publish separate tokens for separate jobs. Treat each one as its own decision.

TokenOwnerPurpose
GPTBotOpenAIContent that may train foundation models
OAI-SearchBotOpenAIChatGPT search
ClaudeBotAnthropicContent that may train Claude models
Claude-SearchBotAnthropicClaude search results
Google-ExtendedGoogleGemini training and grounding, with no effect on Google Search

Google-Extended is a control token only. Crawling still runs on Google’s usual user agents. AI Overviews draw on pages that are indexed and eligible for a snippet, so Googlebot rules govern them.

For bot-by-bot detail, see OAI-SearchBot, ClaudeBot and Claude-SearchBot and Googlebot and Google-Extended. Robots.txt only grants or denies access. Guidance about which pages matter belongs in a separate proposal, llms.txt.

Robots.txt vs noindex

Robots.txt controls crawling. A noindex rule controls indexing. They solve different problems.

Google says a page blocked in robots.txt can still be indexed if other sites link to it, and the result then shows without a description. A crawler that can’t fetch the page also never sees its noindex tag.

To remove a page from search, allow crawling and add noindex. A robots.txt block hides the noindex tag from the crawler that needs to read it.

Common robots.txt mistakes

  • A Disallow: / left over from a staging site.
  • Blocking the CSS and JavaScript files that pages need to render.
  • Adding a named group and forgetting that the bot then ignores the * group.
  • Listing private URLs in the file. Anyone can read robots.txt, so it points people to them.
  • Blocking OAI-SearchBot or Claude-SearchBot while trying to stop training.
  • Fixing the www file and forgetting the subdomains.

How to check and monitor robots.txt

Start with the robots.txt report in Google Search Console. It shows the files Google found for your top hosts, when each was last crawled and any parse errors. You can request a recrawl after a fix.

Robots.txt is also only one layer. A CDN rule or firewall can still block a bot the file allows, and AI crawlers have no equivalent console. Check server logs for each bot’s token and response codes, and recheck the file after every deploy. Expect up to a day before crawlers notice a change, since they may cache the file for 24 hours.

Verify tokens at the source, too. In Ansvisor tracking data, September 15 to October 14, 2026, a GitHub list that includes AI crawler user agents drew 665 citations in AI answers. OpenAI’s own crawler page drew 239. Lists go stale, so confirm each token in the vendor’s docs before you ship a rule.

Our walkthrough on checking whether AI crawlers can access your website covers the log checks step by step.

How Ansvisor helps

Ansvisor’s Site Audit checks robots.txt, AI bot access and llms.txt for any public URL, as part of 47 weighted AEO and GEO signals. Once bots can reach your pages, Citations Monitoring shows which of your URLs AI answers cite and which competitor pages they cite instead.

Robots Exclusion Protocol, REP, Robots.txt File, Robots File, RFC 9309, Robots Exclusion Standard

Let AI Search Work for Your Growth

Each prompt you win brings more mentions, more citations and more buyers. Ansvisor keeps finding the next one.

AI Answers Mentions Citations Visibility Score AI Share of Voice AI Traffic Revenue
Win Back Search VisibilityTry Ansvisor 14-day Free →✓ Zero risk. Cancel anytime.

FAQ

Frequently asked questions.

What is robots.txt?

Robots.txt is a text file at the root of a site, such as example.com/robots.txt. It lists rules that tell crawlers which paths they may fetch. Compliant crawlers read it before crawling, but it isn’t a security control.

What is the Robots Exclusion Protocol?

The Robots Exclusion Protocol is the standard behind robots.txt. Martijn Koster defined it in 1994, and the IETF published it as RFC 9309 in September 2022. It sets the file location, the rule syntax and how crawlers handle errors.

How do I create a robots.txt file?

Create a plain text file named robots.txt, add a User-agent line followed by Allow or Disallow rules, and add your sitemap URL. Upload it to the root of each host, then check Search Console’s robots.txt report for errors.

How do I block AI crawlers with robots.txt?

Add a group for each bot’s token, such as GPTBot or ClaudeBot, with Disallow: /. Check each vendor’s docs first, since search and training bots are usually separate. Blocking a search bot like OAI-SearchBot removes you from that platform’s search answers.

Is robots.txt the same as noindex?

No. Robots.txt controls crawling, and noindex controls indexing. Google says a page blocked in robots.txt can still be indexed if other sites link to it. To keep a page out of results, allow crawling and use noindex.

Help us grow the AI Visibility Glossary

New terms are added regularly.

Help us improve the page or suggest a new term →
About the Author
Cihan Geyik

Cihan Geyik

Co-founder at Ansvisor

Cihan Geyik is the co-founder of Ansvisor, an open-source AI Visibility platform for AI Search. With more than 15 years of experience in digital marketing and growth, he writes about AI visibility, AI search, AEO, GEO, citations, and answer engines. He focuses on helping brands understand and improve their presence across ChatGPT, Gemini, Google AI Overviews, Bing AI Mode and other AI answer engines.

Google Preferred Sources
Get more Ansvisor insights in Google

Add Ansvisor as a preferred source and Google will show you more of our AI search, AEO and GEO guides.