llms.txt Generator and Validator: What the File Does and Does Not Do

Everything runs in your browser. Nothing you type or paste is uploaded, and there is no URL fetch, so paste a URL list or a sitemap body to fill the link rows. Links default to the page URL exactly as you enter it. Switch to .md links only when those Markdown files exist.

The output has no cap on sections, links or length. Titles with square brackets and URLs with parentheses or spaces are repaired, because the reference parser breaks on them. Heading marks in the notes block are removed, because the spec allows no headings there.

Only the H1 is required. Everything else is optional, so decide how much structure your agents need. The file is a proposal by Jeremy Howard, first published in September 2024, not an IETF or W3C standard.

The order is fixed: an optional byte order mark, the H1 with the project name, a blockquote summary, zero or more Markdown sections that contain no headings, then zero or more H2 sections that each hold a list of links. Each list item is a Markdown hyperlink, optionally followed by a colon and notes.

# Acme Docs

> Acme is a billing API for marketplaces. These pages cover setup, the API and release history.

Use the API reference for endpoint details. Quickstart assumes a test-mode key.

## Docs

- [Quickstart](https://acme.test/docs/quickstart): Send your first charge in test mode.
- [API reference](https://acme.test/docs/api): Every endpoint, parameter and error code.

## Optional

- [Changelog](https://acme.test/changelog): Release history, safe to skip for short context.

A file covers the URLs under its path, and where more than one file applies, agents should use the most specific one. That lets a docs subfolder carry its own file next to a root file.

The Optional section is a convention for secondary links an agent can skip when it needs a shorter context. In version 2 it no longer carries mechanical meaning, because the context-expansion tooling left the proposal.

The spec recommends a clean Markdown version of each page at the same URL with .md appended (page.html.md) or the extension replaced (page.md). Version 2 adds link relations so a client can find files without guessing: rel alternate with type text/markdown for a page's Markdown version, and rel describedby for the llms.txt that covers it, sent in an HTML link element or an HTTP Link header.

llms.txt is not robots.txt and not a sitemap. robots.txt signals which access is acceptable. llms.txt is used on demand, when an agent needs information about a topic. The spec says a sitemap usually covers documents too large for a context window.

Do not publish llms.txt to rank in Google. Google's Search Central guide on generative AI features says you do not need llms.txt, AI text files, special markup or Markdown to appear in Google Search, because Google Search itself does not use them.

The same guide says creating the file for other services is fine, and that doing so will neither harm nor help your visibility or rankings in Google Search. It also lists creating unnecessary AI text files such as llms.txt among the AEO and GEO tactics Google Search users can ignore. Google may still crawl and index a text file, but that is not special treatment.

To watch how your content performs in generative AI features, the guide points to the Generative AI performance report in Search Console, which does not depend on any llms.txt file.

John Mueller, a Google Search Advocate, said on his personal Bluesky account in June 2025 that no AI system currently uses llms.txt, and that the consumer chatbots fetch your pages but not the file.These are his own posts, not Google documentation.

In March 2026 he explained why log tests mislead: many sites link to every known llms.txt file, so crawlers reach them like any text file, and a self-linked cats.txt could be just as effective. In September 2026 he repeated that such tests work only because directory sites link to every file they find.

Read every log study as a count of requests, not of use. A fetch does not mean the content reached an answer, and Ahrefs calls every figure in its study a ceiling on actual consumption.

The studies disagree on whether any AI crawler reads the file. EZY reports that Meta-ExternalAgent fetched llms.txt more often than robots.txt, and Longato logged OpenAI agents hitting it. Ahrefs, Otterly and Buytaert see almost nothing from answer engines. The consistent part is small counts concentrated in coding agents, SEO tools and a few crawlers, with no study showing that any of it reaches answers.

Adoption figures differ because the samples differ. Ahrefs reports that 28% of the domains in its study publish a file, and says that is an upper bound because its customers skew technical.SE Ranking reports 10.13% of nearly 300,000 domains. Longato probed 4,685 hosts and only 137 returned a 200 at /llms.txt. Do not average them.

Ahrefs saw zero AI-bot requests to llms.txt URLs that returned an error. It reads this as AI tools fetching the file only when something points to it.Ahrefs did not test whether the files were well formed against the spec, which is one reason to run the validator above on your own file.

Control AI crawlers with robots.txt, because that is the file their documentation names. None of the three crawler pages tells site owners to publish llms.txt, and none says its crawlers or answer products read the file on third-party sites. That is an absence on those pages, not a statement that they ignore it.

The pattern is consistent: the labs publish llms.txt where developers and coding agents read documentation, and not on their marketing sites. The spec page makes the same claim about OpenAI, Anthropic and Gemini.Copy that placement: put the file where an agent goes to learn your product, and keep AI-crawler access rules in your technical SEO checklist under robots.txt.

Publish the file if people use agents to work with your product. The spec author says llms.txt files are used most heavily for software documentation, where coding agents follow them to API references and tutorials, and that the structure also fits a business, a personal site or a school.He also says coding agents use them reliably, which is his statement and not a measurement.

The one independent signal points the same way. In the Ahrefs data, Claude-Code, Anthropic's coding agent, out-fetched every AI retrieval bot, every AI assistant and every AI training crawler except statespace-indexer and GPTBot.Ahrefs' verdict is that for visibility in ChatGPT, Perplexity or AI Overviews the file is largely decoration, and that the nearest thing to an intended audience is coding agents. Longato's advice is to create it because it is cheap, not because it will get you cited in ChatGPT today.

Documentation platforms already do the work. Mintlify generates and hosts llms.txt and llms-full.txt, serves the file at /.well-known/llms.txt too, and adds a Link header and an X-Llms-Txt header to every response. Its links carry a .md extension so tools fetch the Markdown page directly, and it cuts descriptions taken from page frontmatter at 300 characters.The llms-full.txt file is not defined on the spec page. It is a convention from documentation platforms.

For a marketing site with no docs, the file is a small task and carries no ranking upside. If you want an agent-facing interface that is actually in use, the GrowthVector MCP server shows what a structured alternative looks like, and the Claude Code best practices page covers how coding agents consume project context.

Three checkers disagree on what a valid file is, so the validator labels each message with its source. A file can pass Lighthouse and still fail the spec or the reference parser, so fix errors first and treat warnings as judgment calls.

Lighthouse is the loosest checker. In its source, the pass rules are an H1 matched by a multiline pattern, at least one link, and a length of at least 50 characters. It does not check Content-Type, line order, the blockquote, H2 sections or whether links resolve.It also fetches only /llms.txt at the root of the final displayed URL, never a subpath file.

The reference parser from the spec author's organization is the strictest on link syntax. It needs a leading hyphen, a title without a closing bracket and a URL without a closing parenthesis, and it expects the H1 first, then the blockquote, then H2 sections.

Measure size as information only. The spec sets no limit and asks only that the file fit in context, so the tool reports bytes, characters and an estimated token count and never blocks or trims the output. For a larger site, the Mintlify approach of splitting an index above 100,000 characters into nested index files is one model.

Serve the file at /llms.txt and confirm the response, not just the editor preview. The spec says nothing about Content-Type, encoding, size or caching. In practice, OpenAI, Anthropic, Perplexity and GrowthVector serve text/plain with a UTF-8 charset, and Google's Gemini docs serve text/markdown. Use text/plain; charset=utf-8 unless you have a reason not to.

curl -sI https://example.com/llms.txt
# Expect: HTTP 200 and Content-Type: text/plain; charset=utf-8
curl -s https://example.com/llms.txt | head -5
# Expect: the H1 on line 1, not an HTML page

The soft 200 is the failure to test for. A catch-all route that returns your 404 page as HTML with a 200 status passes a status check and fails every parser. In his crawl of 4,685 hosts, Longato found 137 with a 200 at /llms.txt, 111 of them as text/plain, and only 20 with real content. 23 of 175 responses with a 200 were HTML (175 counts crawl rows, 137 counts distinct hosts).

Do not trust a single tool's error message. A Lighthouse issue closed in June 2026 shows the audit once reported "Fetch of llms.txt failed" for a spec-compliant file that Google's PageSpeed Insights fetched with a 200. A maintainer said a later version would display the error more clearly.Confirm with curl, then paste the exact body into the validator.

Make the file discoverable. Agents fetch it when something points to it, so send the link relation from the pages it describes.

Link: </llms.txt>; rel="describedby", </docs/page.md>; rel="alternate"; type="text/markdown"

GrowthVector's own llms.txt is a worked example of the plain version. It has an H1, a blockquote summary and H2 sections of links to HTML pages, and no Optional section or .md page versions, so it makes no claim to the Markdown convention. To see whether any site publishes a file, run the SEO Oracle audit, which reports whether llms.txt is present with its section and link counts.

Keep the file a factual map of real pages, and regenerate it when the site changes. A stale file misleads every agent that reads it, and Ahrefs lists prompt injection through a stale or compromised file as a risk. That framing is Ahrefs' opinion, but the observation behind it is real: the largest single research crawler in its data identified itself as prompt-injection-survey/1.0.

The academic basis for manipulation concern is a 2024 paper showing that crafted website content can steer LLM search engines, tested on Bing and Perplexity, and plugin selection for GPT-4 and Claude. The paper does not study llms.txt.The practical rule follows: never put claims in the file that the visible page does not support. Mueller joked about sites writing into llms.txt that they are the top authority in their field, and a model that reads the visible page will see the gap.

Resist building a separate agent-only copy of your site. Mueller wrote that making a separate agent-friendly version to fix accessibility issues is building technical debt, and that LLMs have always read and parsed normal web pages, so HTML is not a problem for them.

Generate the file from your sitemap or CMS in the build, keep it in version control and review the diff on release. For the work that does move AI visibility, read the GEO and AEO guide, the overview of how search works across Google and AI platforms and the SEO guide.