llms.txt is a markdown file placed at a site’s root that gives AI agents a short guide to your most useful pages. It was proposed in 2024 and is not part of any web standard. Google says Search does not use it. The crawler documentation from OpenAI, Anthropic, and Perplexity does not describe using it for ranking or citations, though OpenAI and Perplexity publish one for their own docs.
What the proposal says
The proposal is hosted at llmstxt.org and credited to Jeremy Howard, dated Sep 3, 2024. The version now online is labeled v2. Its premise is that agents use websites constantly, that HTML pages are wrapped in navigation, ads, and scripts, and that agents are better served by concise information gathered in one place.
The format is specific. The file lives at /llms.txt or at a subpath, and it covers the URLs under that path. It contains, in order:
- An H1 with the name of the site or project. This is the only required part.
- A blockquote with a short summary.
- Optional paragraphs or lists with more detail, but no headings.
- H2 sections that hold lists of links, each written as a markdown link with optional notes.
By convention, a section titled “Optional” marks links an agent can skip when it needs a shorter context. The proposal also suggests publishing a clean markdown version of each page at the same URL with .md added, and pointing to it with a rel="alternate" link.
It states how it differs from existing files. A sitemap lists every indexable page and does not point to markdown versions. robots.txt tells automated tools what access is acceptable. llms.txt is for on-demand use, when an agent needs information to help a person, and the proposal says it was expected to be useful mainly for answering questions, not for training.
The proposal says thousands of sites publish the file, that documentation platforms generate it automatically, and that OpenAI, Anthropic, and Gemini publish one for their own developer docs. You can see the first claim for yourself. The OpenAI and Perplexity crawler pages both open with a pointer to their own llms.txt documentation index.
What Google has said
Google’s guide to optimizing for generative AI features was published in May 2026. Under a “mythbusting” heading it lists llms.txt files and other “special” markup as something you can ignore, saying you do not need machine-readable files, AI text files, markup, or Markdown to appear in Google Search, including its generative AI features, because Google Search does not use them.
Google’s update log shows it came back to the point on Jun 15, 2026, adding a note to clarify. The note says it is fine to create and maintain llms.txt files for other services that use them, and that doing so will neither harm nor help your visibility or rankings, because Google Search ignores them.
Search Engine Journal reported that this matches earlier statements. According to its May 20, 2026 article, John Mueller compared llms.txt to the keywords meta tag, saying no AI services used it and bots did not request the file, and Gary Illyes and a colleague said at a Search Central Live event in Asia Pacific that Google was not pursuing it. Treat those as reported statements. The guide is the primary source.
Where Google’s own products disagree
The same Search Engine Journal article points out an odd split. Around the time the guide came out, Chrome’s Lighthouse 13.3 added an Agentic Browsing category that includes an llms.txt audit. Lighthouse describes the file as a machine-readable summary meant for AI agents and says that without it, agents may spend more time crawling the site to understand its structure.
Those 2 positions are consistent once you separate the jobs. The Search guide is about appearing in Google Search. The Lighthouse check is about browser-based agents understanding a site, and the article notes it is separate from the SEO audits. Neither claims the file improves rankings or citations.
What this means for a business site
Our view, based on the documentation above:
- Search visibility. Do not expect anything from Google. For other engines, there is no documented ranking or citation use that we found in the vendor pages reviewed for this article.
- Cost. A short file takes an hour. Keeping it accurate is the real cost, because a stale file that points agents at dead pages is worse than none.
- Where it makes sense. Documentation, API references, and product help content, where coding assistants and agents fetch pages to answer specific technical questions. This is the use the proposal describes most.
- Where it does not. A brochure site with 12 pages gets little from a curated list of 12 pages.
If you add one, keep it small, link to your best pages, and put a review date in your calendar. Do not rewrite your content for it, and do not use it as a substitute for the work in our guide on writing pages AI engines cite. The step-by-step version is in the llms.txt guide.
It does not block or allow anything
A common mix-up is treating llms.txt as a permission file. It is not. It does not stop any bot from reading your pages, and it does not grant access either. Those decisions stay in robots.txt, in your firewall, and in the page-level controls each company documents. If your goal is to keep content out of model training, our guide to AI crawlers and robots.txt covers the right switches. llms.txt only describes what you would like an agent to read first.
What a minimal file looks like
Here is an example for a made-up business. The name and links are invented for illustration.
# Example Plumbing Co.
> Licensed plumbing and heating contractor serving 3 counties. Emergency service, installs, and repairs.
## Services
- [Emergency repairs](https://example.com/emergency): Same-day service area and hours
- [Water heater installs](https://example.com/water-heaters): Options and what an install involves
## Optional
- [About us](https://example.com/about)Notice how little there is. The value of the file, if there is any, comes from picking the 5 to 10 pages an agent should read first and describing each in a line. A file that lists every URL on the site is a sitemap with extra steps.
How to tell whether anyone reads it
Check your server logs for requests to /llms.txt and see which user agents make them. If you publish one and no AI crawler ever asks for it, that tells you what the file is currently worth on your site. Search Engine Journal reports that Mueller’s keywords meta tag comparison rested on bots not requesting the file. The test costs nothing and settles the question for your own domain.
