llms.txt is a proposed, voluntary standard for publishing a clean Markdown map of your key content so large language models can read it efficiently. As of 2026, no major AI crawler — Google, OpenAI, Anthropic, or Perplexity — has confirmed it uses llms.txt to rank, retrieve, or cite pages, and server logs suggest it is rarely fetched at scale.
That makes adding one low-cost and low-risk, but it is not a ranking signal, a citation guarantee, or a substitute for the fundamentals. The clearest payoff today is narrow and real: coding agents and developers who paste your documentation into a context window. If you run a docs-heavy or developer-tool SaaS, ship it. If you are hoping it will rescue your visibility in AI answers, fix crawlability, structured content, and schema first.
That verdict runs against a lot of the confident advice circulating in 2026. This guide walks through what llms.txt actually is, the exact format, what problem it claims to solve, and — the part most articles skip — the evidence on whether anything reads it. We attribute the adoption and skepticism claims to named sources with dates, because on this topic the gap between marketing enthusiasm and observable behavior is wide.
What llms.txt is, precisely
llms.txt is a plain-text file, written in Markdown, that you place at the root of your domain — for example, at yourdomain.com/llms.txt. It was proposed by Jeremy Howard, co-founder of Answer.AI, and first published on September 3, 2024. The specification lives at llmstxt.org, which has since been revised based on adoption experience.
The premise is simple. When a language model tries to use your website, it hits two problems. First, HTML pages are noisy: navigation menus, cookie banners, JavaScript, ads, and boilerplate surround the actual content, and models have to spend tokens parsing all of it. Second, context windows are finite. Even large ones fill up fast when a model is asked to reason over a sprawling site. llms.txt tries to solve both by giving the model a short, human-curated index that points to the clean, authoritative versions of your key pages.
It is important to be precise about what kind of file this is. llms.txt is not an access-control file. It does not tell crawlers what they may or may not fetch, the way robots.txt does. It does not add machine-readable metadata to a page the way schema markup does. It is closer to a curated table of contents written for a machine reader — a suggestion, not a directive, and one that a model is free to ignore.
The exact format
The specification defines a strict but small structure. In order, an llms.txt file contains:
- An H1 heading — the name of the project or site. This is the only required element.
- A blockquote — a short summary containing the key information needed to understand the rest of the file. Optional but strongly recommended.
- Zero or more sections of detail — paragraphs and lists giving more context. These sections must not use headings, to keep parsing predictable.
- H2-delimited “file list” sections — each H2 introduces a list of links. Every item is a Markdown hyperlink in the form [name](url), optionally followed by a colon and a short note describing the file.
By convention, a section titled “Optional” signals content a model can safely skip when its context budget is tight. The whole design leans on Markdown precisely because it is cheap for a model to parse and easy for a human to maintain by hand. A minimal example, described in prose since code blocks are not used here: an H1 reading “Acme API,” a blockquote summarizing that Acme is a payments API with REST and webhook interfaces, then an H2 “Docs” section listing links such as a quickstart, an authentication guide, and an API reference, each with a one-line note. That is the entire file. It is meant to be small.
llms.txt versus llms-full.txt
Two files are often discussed together, and they do different jobs. llms.txt is the curated index described above — a map of links for a model to navigate. llms-full.txt is different: it concatenates the full text of your documentation into a single Markdown file, so a developer or agent can drop one URL into a context window and get everything at once. Mintlify, a documentation platform that auto-generates both, describes llms-full.txt as compiling all of a site’s text into one file for complete-context use.
The distinction matters for how you think about payoff. llms.txt is a curation-for-navigation artifact. llms-full.txt is a bulk-context artifact. The second is the one developers actually paste into ChatGPT or Claude when they want the model to answer questions about a tool — which, as we will see, is where the most defensible real-world usage sits today.
What problem it claims to solve
Proponents frame llms.txt as inference-time help rather than a crawling instruction. The argument runs like this: models increasingly answer questions by retrieving and reading pages on demand, not just from what they memorized during training. If you hand the model a clean, curated map, it spends fewer tokens on parsing, is less likely to be confused by boilerplate, and is more likely to land on the page you actually want it to read. Advocates also point to a bandwidth benefit — directing a model to a few clean Markdown files instead of forcing it to crawl and render your entire site.
Notice what is being claimed and what is not. The honest version of the case is about comprehension and efficiency — helping a model that has already decided to read your content read it better. It is not, in its careful form, a claim that llms.txt will make a model choose your site over a competitor’s, or cite you more often. That stronger claim is where the evidence falls apart, and where a lot of 2026 marketing quietly overreaches.
The evidence: does anything actually read it?
This is the section that should decide your effort, and it is the one most guides gloss over. The short answer: the major web-scale AI crawlers give no public confirmation they use llms.txt, and the observable data suggests they largely do not fetch it. The clearest usage is a narrower category — tools and agents that read the file deliberately.
The skeptical case, from named sources
The most-cited skeptic is Google’s John Mueller. In an April 2025 discussion, responding to a site owner who reported that no AI bots were downloading their llms.txt file, Mueller wrote: “AFAIK none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it). To me, it’s comparable to the keywords meta tag.” The comparison is pointed. The keywords meta tag was an early-web feature that let sites declare their own relevance; it was abused so thoroughly that search engines stopped trusting it entirely. Mueller’s implication is that a self-declared, self-curated file is structurally the kind of signal a serious retrieval system learns to ignore, because content the model can already fetch directly is more trustworthy than a summary the site wrote about itself.
The server-log evidence supports the “nobody’s fetching it” half of that claim. In the same thread, a commenter hosting roughly 20,000 domains reported no meaningful bot activity against these files. Broader analyses point the same direction. SEOmator’s 2026 GEO Data Report, which analyzed roughly 515 million bot events, found crawl-to-referral ratios so lopsided — on the order of thousands of pages fetched per single human referral for GPTBot and ClaudeBot — that self-declared files like /llms.txt barely register against the volume of ordinary page fetches. Google’s own Search guidance, as of its May 2026 revision, does not mention llms.txt at all. And a controlled analysis by Trakkr reportedly found no statistically significant relationship between having an llms.txt file and being cited by AI answer engines (a p-value around 0.85, which is about as close to “no effect detected” as these things get).
The case for, kept honest
The counterargument deserves a fair hearing, because the meta-keywords comparison is not perfect. Writing in Search Engine Land, practitioners have argued that llms.txt differs in one structural way: it curates a list of real URLs, and the linked content has to exist and deliver when the model arrives. You cannot stuff it with unverifiable keyword claims the way the old meta tag allowed. In that framing it is less like metadata and more like a curated storefront — useful precisely because the model can verify what it points to.
More concretely, there is one consumer category that demonstrably reads these files: coding agents and developer tools. Documentation platforms report that AI agents visit llms.txt and, more often, llms-full.txt, because a developer using an assistant like Cursor or Windsurf is effectively asking the tool to load a library’s docs into context. Mintlify has cited internal and third-party benchmarks — including a LangChain comparison — suggesting agents given a curated llms.txt outperformed unaided retrieval on documentation tasks. Treat vendor benchmarks with appropriate caution, since the vendors sell the tooling, but the underlying mechanism is plausible and matches how developers actually work: they paste a docs URL into a chat window on purpose.
So the two halves reconcile cleanly. Web-scale, passive AI crawlers — the ones that feed AI Overviews and general chatbot answers — show little sign of using llms.txt in 2026. Deliberate, tool-driven consumption by coding agents and developers is real. Whether your site benefits depends entirely on which of those audiences matters to you.
How llms.txt differs from robots.txt, sitemaps, and schema
A lot of confusion comes from treating llms.txt as if it were a proven web standard on par with the files it superficially resembles. It is not, and the difference in status is the whole point.
| File or standard | What it does | Who reads it | Proven in 2026? |
|---|---|---|---|
| robots.txt | Controls which crawlers may access which paths | Search and AI crawlers, widely respected by major operators | Yes — a decades-old, honored standard |
| sitemap.xml | Lists your URLs to aid discovery and crawling | Search engine crawlers | Yes — officially supported by Google and Bing |
| Schema / structured data | Adds machine-readable meaning to page content | Search engines; used for rich results and, increasingly, AI answers | Yes — documented and used in ranking features |
| llms.txt | Offers a curated Markdown map of key content for LLMs | Some coding agents and developers; not confirmed by major AI crawlers | No — a voluntary proposal, unconfirmed by major vendors |
The practical reading of this table: robots.txt, sitemaps, and schema are things a serious B2B site should have because we know they are consumed and, in the case of schema, increasingly implicated in how AI systems parse pages. llms.txt sits in a different category — a reasonable bet with a thin evidence base, worth making only after the proven work is done. If you want the deeper argument for why structured, machine-readable content is the durable investment, our writing on generative engine optimization lays out where the returns actually come from.
Who should bother — and who should not
The right answer is not universal, which is exactly why blanket “every site needs llms.txt” advice should make you suspicious. Match the decision to your audience and content type.
Good candidates
- Developer-tool and API companies. If your buyers evaluate you by reading docs, and their engineers use AI coding assistants, llms.txt and especially llms-full.txt serve a real, present-day workflow. This is the strongest case, and it is no accident that the visible adopters skew heavily toward this group.
- Docs-heavy SaaS. Products with large, structured documentation libraries get the most from a curated map, because there is a real navigation problem to solve.
- Sites already doing the fundamentals well. If your crawlability, structured content, and schema are in good shape, llms.txt is a cheap incremental experiment rather than a distraction.
Weak candidates
- Thin or marketing-only sites. If you have a dozen pages and no documentation, there is little to curate and less to gain. Your effort belongs in creating substantive content first.
- Sites hoping to fix AI visibility with one file. If your pages are not being crawled, cited, or rendered well, llms.txt changes none of that. It is a map to content, not a fix for content.
- Teams with limited time. If you can do one thing this quarter, it is not this. The expected value of getting schema and crawlability right is far higher and far better evidenced.
Reported adoption numbers reinforce the point about who finds it worthwhile. According to a study reported by PPC Land in 2026, only about 7.4% of Fortune 500 companies had shipped an llms.txt file, and developer-documentation sites adopted markedly faster than everyone else. Named adopters frequently cited include Anthropic, Cloudflare, Stripe, Vercel, Supabase, Zapier, and several open-source frameworks — a roster that is overwhelmingly developer-facing. That is a signal about fit, not about universal necessity. Do not read a docs platform’s enthusiasm as proof your marketing site needs the same thing.
If you decide to add one: a step-by-step
Adding llms.txt is low-effort, which is the best argument for doing it once the fundamentals are handled. Here is a sensible sequence.
- Confirm the fundamentals are in place first. Check that your important pages are crawlable, that your sitemap is current, and that your key templates carry valid schema. If any of that is broken, stop and fix it before touching llms.txt.
- Inventory the content worth surfacing. List the pages a model would most need to understand what you do: core docs, key product pages, pricing, an about page, primary guides. Keep the list tight. Curation is the entire value.
- Write the file by hand or generate it. Start with an H1 site name, a one- or two-sentence blockquote summary, and then H2 sections grouping your links with short notes. If you run documentation on a platform that auto-generates the file, let it — but review the output rather than trusting it blindly.
- Consider llms-full.txt separately. If you have real documentation and a developer audience, publish a full-text version too. This is the file that gets pasted into context windows, so for dev-tool companies it may matter more than the index.
- Publish at the root. Place llms.txt at yourdomain.com/llms.txt so it is discoverable at the conventional location. Serve it as plain text or Markdown.
- Keep it accurate and current. A stale map is worse than none, because the one thing that distinguishes llms.txt from the discredited meta-keywords tag is that its links must actually exist and deliver. Update it when your content changes.
- Measure with discipline. Watch your server logs for requests to the file, and watch which agents make them. Do not expect ranking movement, and do not attribute unrelated traffic to the file. If nothing ever fetches it, that is data too.
Total time for a modest site is an afternoon. That low cost is precisely why it can be worth doing even on thin evidence — and precisely why it should never crowd out the work that is proven to matter.
The fundamentals that matter more
If llms.txt is a speculative bet at the margin, what is the durable work? The same disciplines that make you findable in classic search increasingly govern whether AI systems can read, trust, and cite you — and unlike llms.txt, these are documented and observable.
Crawlability and clean rendering. AI systems that fetch pages on demand still have to reach and parse your content. If key pages are blocked, buried behind heavy JavaScript, or slow, no curated map compensates. This is the floor, and it is where more B2B sites fail than they realize.
Structured, extractable content. Clear headings, direct answers near the top of the page, and self-contained sections make your content easy for a model to lift and quote. This is the mechanism behind being surfaced in AI answers, and it is the throughline of our work on answer engine optimization and on optimizing content for the prompts people actually ask.
Schema markup. Structured data is a proven, vendor-supported way to tell machines what your content means. It is consumed today, unlike llms.txt, and it compounds across both traditional and AI-driven surfaces.
Real citations and authority. Being referenced by sources AI systems already trust — and maintaining an accurate presence on the open, structured platforms models retrieve from — does more for your visibility than any file you host yourself. When your site blocks or breaks, those external signals are what remain retrievable.
Defending existing visibility. As AI Overviews and answer boxes absorb clicks, protecting the queries you already win becomes its own discipline — one we cover in our guide to surviving Google AI Overviews. That is where the revenue risk is real and present, not in whether you shipped a Markdown index.
The pattern across all of these is that they change what a machine can observe about your site, and their effects are measurable. llms.txt asks a machine to trust a summary you wrote about yourself, and its effects, so far, are not. That is the difference between infrastructure and a hopeful gesture. Build the infrastructure first. If you want help sorting which of these actually moves revenue for your accounts, that is the core of our AI search and SEO work, and you can always talk to us about where llms.txt fits in a real program — usually near the bottom, after the proven work is done.
The verdict
llms.txt is a thoughtful proposal solving a plausible problem, championed by credible people, and adopted by a specific, developer-heavy slice of the web. It is also, as of 2026, unconfirmed by every major AI crawler, largely unfetched at web scale according to server-log evidence, dismissed by Google’s own search advocate as comparable to the keywords meta tag, and unlinked to citations in the controlled testing that exists. Both of those things are true at once, and the resolution is not to pick a side but to match the tool to the job. If you are a docs-heavy or developer-tool SaaS whose buyers live in AI coding assistants, ship llms.txt and llms-full.txt — the payoff is narrow but real. For everyone else, treat it as a cheap, optional experiment to run after the fundamentals, never instead of them. It will not hurt you. Just do not expect it to be the thing that gets you found.
Frequently asked questions
What is llms.txt?
llms.txt is a proposed, voluntary standard — a plain-text Markdown file placed at your domain’s root (yourdomain.com/llms.txt) that gives large language models a clean, curated map of your most important content. It was proposed by Jeremy Howard of Answer.AI in September 2024. It is a suggestion a model can read, not an access-control file like robots.txt, and as of 2026 no major AI crawler has confirmed it uses one.
Does Google or ChatGPT actually use llms.txt?
No major AI vendor — Google, OpenAI, Anthropic, or Perplexity — has publicly confirmed using llms.txt to rank, retrieve, or cite pages. Google’s John Mueller stated in April 2025 that none of the AI services had said they use it and that server logs show they do not even check for it, comparing it to the discredited keywords meta tag. Server-log studies support that these files are rarely fetched at web scale. The exception is deliberate use by coding agents and developers who load documentation into a context window.
Does llms.txt help SEO or AI rankings?
There is no evidence it does. Google’s May 2026 search guidance does not mention it, and a controlled analysis by Trakkr found no statistically significant link between having an llms.txt file and being cited by AI answer engines. Treat it as neutral for rankings. The proven levers for AI visibility remain crawlability, structured and extractable content, schema markup, and real external citations.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a curated index — an H1 site name, a short summary blockquote, and lists of links to your key pages with brief notes. llms-full.txt is different: it compiles the full text of your documentation into one Markdown file, so a developer or agent can paste a single URL into a context window and load everything at once. For developer-tool companies, llms-full.txt is often the more useful of the two.
Should my B2B site add an llms.txt file?
Only after the fundamentals are handled, and mainly if you are a docs-heavy or developer-tool SaaS whose buyers use AI coding assistants — that is where the real, present-day payoff sits. It is cheap to add (an afternoon) and low-risk, so it can be a reasonable experiment. But if your crawlability, structured content, and schema need work, spend your time there first; llms.txt is a map to content, not a fix for it.