
A coding agent that needs your API docs, or an assistant answering a question about your pricing, has to find the right page fast. HTML full of menus, scripts and cookie banners slows that down. A file called llms.txt offers a shortcut: one Markdown file that says what the site is and where the useful content lives. It’s cheap to publish, and it’s easy to overrate.
llms.txt is a file format proposed by Jeremy Howard in September 2024. The llmstxt.org proposal describes its purpose as providing information to help agents use a website.
The file sits at /llms.txt, either at the site root or under a subpath such as /docs/llms.txt. When more than one file applies, the current version of the proposal says agents should use the most specific one.
The proposal also asks sites to offer Markdown versions of useful pages at the same URL with .md added. Agents can then read clean text without parsing HTML.
The format is plain Markdown with a fixed order. Only the H1 is required.
| File | Who reads it | What it does |
|---|---|---|
| robots.txt | Crawlers, before fetching | Grants or denies access to paths |
| sitemap.xml | Search engine crawlers | Lists every URL you want found |
| llms.txt | AI agents and LLM tools, on demand | Gives a curated summary and links to the best content |
The difference matters. llms.txt can’t block a bot or invite one in. A sitemap lists everything, while llms.txt picks the few pages that explain the site best.
The proposal’s site says thousands of sites publish the file. They include the developer docs of OpenAI, Anthropic and Google’s Gemini.
Tooling has followed. Documentation platforms such as Mintlify and GitBook generate the file, as do SEO plugins like Yoast SEO and AIOSEO, and Wix. The proposal also notes that Chrome’s Lighthouse checks for one in its agentic browsing audits.
We publish one at ansvisor.com/llms.txt. It has 11 sections, from platform features to the glossary, with short notes on each link.
There’s no public evidence that it does. Google’s guide to optimizing for generative AI search is direct about it. You don’t need new machine-readable files, AI text files or Markdown to appear in Google Search, because Search doesn’t use them.
That rules out a direct effect on AI Overviews and AI Mode. For other platforms, no vendor has said it ranks or cites pages based on llms.txt.
The realistic benefit is narrower. AI agents and coding tools that fetch your docs can find the right page with fewer steps. For a developer product, that’s worth an hour of work. For a local bakery, it’s optional.
Several open-source projects generate llms.txt along with JSON-LD and robots.txt rules. Our list of open-source AEO and GEO tools for developers covers a few.
Ansvisor’s Site Audit checks for an llms.txt file alongside robots.txt and AI bot access, as part of 47 weighted signals for any public URL. The file counts as one signal among many, which matches the evidence above.
Each prompt you win brings more mentions, more citations and more buyers. Ansvisor keeps finding the next one.
llms.txt is a proposed file, written in Markdown and placed at /llms.txt, that summarizes a website for AI agents. It lists the site’s most useful pages with short notes, so an agent can find answers without reading every page.
No. Jeremy Howard proposed it in September 2024, and llmstxt.org still calls it a proposal open to community input. Thousands of sites publish one, but no standards body has adopted it.
Google says Google Search doesn’t use llms.txt and you don’t need AI text files to appear in its results. There’s no public evidence that it raises citations elsewhere. Its value is helping agents and developer tools read your content.
Start with an H1 naming the site, then a one-line blockquote summary. Add H2 sections with Markdown links to your most useful pages, each with a short note. Put links an agent can skip under a section called Optional.
Robots.txt tells crawlers which URLs they may fetch. llms.txt doesn’t grant or block anything. It’s a curated guide that agents read when they need information about your site.
See how AI answers describe your brand, find out why competitors get picked, and close the gaps with AI agents that work alongside your team.
14-day free trial or 50% off your first month with code SAVE50.
Continue exploring key AI visibility concepts.
Measure and improve how often your brand appears in AI-generated answers.
Learn more →Strategies for increasing visibility in answer engines and AI summaries.
Learn more →Optimizing content for AI-powered discovery experiences.
Learn more →Understand how OpenAI retrieves and synthesizes information.
Learn more →AI-generated summaries that appear directly in Google Search.
Learn more →Explore how Perplexity cites and presents sources.
Learn more →References and sources used by AI systems to support answers.
Learn more →Measure the quality and influence of cited sources.
Learn more →How easily AI systems can discover and reuse your content.
Learn more →New terms are added regularly.
Help us improve the page or suggest a new term →
Co-founder at Ansvisor
Cihan Geyik is the co-founder of Ansvisor, an open-source AI Visibility platform for AI Search. With more than 15 years of experience in digital marketing and growth, he writes about AI visibility, AI search, AEO, GEO, citations, and answer engines. He focuses on helping brands understand and improve their presence across ChatGPT, Gemini, Google AI Overviews, Bing AI Mode and other AI answer engines.
Add Ansvisor as a preferred source and Google will show you more of our AI search, AEO and GEO guides.
© 2026 Ansvisor. All rights reserved. Ansvisor is an open-source AI Search Execution Platform for AI Visibility. Ansvisor is a brand owned and operated by Empler AI Inc.
The open-source AI Search Execution Platform. Track how AI answers talk about your brand and act on it.
Open source · MIT license© 2026 Ansvisor, a brand of Empler AI Inc. · 1401 PE Ave. Unit 105, Wilmington, DE 19806, United States
Cookie preferences
We use cookies to keep the site working and, with your OK, to measure traffic and improve our marketing. Read our Cookies & Tracking policy.