Do you keep hearing that your site “needs an llms.txt file” for AI search? Here is the short answer. llms.txt is a proposed Markdown file that gives AI tools a hand-picked map of your most useful pages. It is easy to make and harmless. But Google Search does not use it, and no major AI search provider has publicly promised to use it either.
So the real question is not “how do I get one” but “is it worth an hour of my time”. In this guide, I explain what the file is, what the format looks like, and who actually uses it as of late September 2026. I also cover how it differs from robots.txt, and which kinds of site should bother.
What Is llms.txt?
llms.txt is a plain text file written in Markdown (a simple way to format text with symbols like # and -). It sits at /llms.txt on your domain. Jeremy Howard, co-founder of Answer.AI, proposed it on 3 September 2024. The proposal is published at llmstxt.org, which describes it as a way to give information that helps agents use a website.
The problem it tries to solve is real. A web page is full of menus, scripts, cookie banners and adverts. A language model can only read a limited amount of text at once, so it wastes a lot of that space on clutter. A short, clean Markdown summary with links to the pages that matter is much easier for a model to work with.
Think of it as a hand-written table of contents for machines. It does not allow or block access to anything.
It Is a Proposal, Not a Standard
No standards body has adopted llms.txt, and no search engine or AI company has to read it. That is the key difference from robots.txt. Robots.txt has been made an official internet standard, and the major search and AI crawlers follow it.
The llms.txt Format Explained
The specification on llmstxt.org is short. The file uses ordinary Markdown, and the sections must appear in this order:
- An H1 with the name of the site or project. This is the only required part.
- A blockquote with a short summary of the key facts someone needs to understand the rest of the file.
- Optional free text, such as paragraphs or bullet lists, with extra context. No headings in this part.
- Zero or more H2 sections that act as “file lists”. Each item is a Markdown link,
[name](url), with an optional colon and short note after it. - An H2 called “Optional”, by convention. Links in this section are less important. Tools can skip them when they need a shorter version.
The file normally lives at the root of the site. But the specification also allows one in a subfolder, such as /docs/llms.txt, covering the URLs below it. When more than one applies, tools are meant to use the most specific file.
Markdown Versions of Your Pages
The proposal also suggests offering a clean Markdown copy of important pages at the same URL with .md added. For example, /pricing.html.md, or /guide/index.html.md for URLs without a file name. The links in your llms.txt can then point to those lighter versions.
The current version of the specification adds a way to point to these files. A page can include a rel="alternate" type="text/markdown" link pointing to its Markdown copy. It can also include a rel="describedby" link pointing to the llms.txt file that covers it. Both can also be sent as an HTTP Link header, which works for PDFs and other non-HTML files.
What About llms-full.txt?
You will often see a second file, llms-full.txt, mentioned next to llms.txt. It is not part of the llmstxt.org specification. It is a companion idea made popular by documentation platforms such as Mintlify, which began creating both files automatically in late 2024.
The difference is simple:
- llms.txt is an index: a summary plus links.
- llms-full.txt is the full text: the content of every listed page joined into one large Markdown file.
llms-full.txt suits documentation, where a developer might paste a whole product manual into an AI coding assistant. For a marketing site or a blog, it adds little and can become very large. If you publish one, make sure it matches the live content and is rebuilt when pages change.
Who Supports llms.txt as of Late September 2026?
This is where most articles on the topic get vague. So I find it helps to separate two things: who publishes an llms.txt file, and who reads and uses other people’s files.
Google’s Position
Google has been consistent, and its statements are on the record:
- June 2025: Google’s John Mueller wrote on Bluesky that no AI system currently uses llms.txt. He said AI chatbots fetching web pages do not fetch the file.
- July 2025: Gary Illyes said at a Google Search Central Live event that Google does not support llms.txt and has no plans to.
- December 2025 to January 2026: llms.txt files appeared on several Google developer sites after a change across their CMS. The file was quickly removed from the Search developer documentation. When asked whether the remaining files were an endorsement, Mueller replied plainly that they were not. He has also compared llms.txt to the old keywords meta tag.
- Google’s own documentation: the Search Central guide on optimising for generative AI features (last updated 10 July 2026) is clear. It says you don’t need to create machine-readable files, AI text files, markup or Markdown to appear in Google Search, because Google Search doesn’t use them. It adds that creating them will neither harm nor help your visibility in Search.
That covers Google Search, including AI Overviews and AI Mode. An llms.txt file will not change how you rank or whether you are cited there.
AI Companies and Other Platforms
Many technology companies publish llms.txt files for their developer documentation, including AI companies such as Anthropic. That shows the file is useful to them as documentation. It does not show that their crawlers or search products rely on llms.txt files from other sites.
As of late September 2026, none of the major AI search providers has said in its crawler or webmaster guidance that it reads llms.txt to decide what to crawl, cite or recommend. Some site owners have reported seeing AI bots request the file in their server logs. But a request is not the same as using it to rank or cite anything.
Tools That Generate the File
Support is strongest among the tools that create the file:
- Yoast SEO added an automatic llms.txt generator in version 25.3, released on 10 June 2025.
- Rank Math offers an llms.txt option you can switch on in the plugin.
- Documentation platforms such as Mintlify create llms.txt and llms-full.txt automatically for hosted docs.
Where the file really gets used is when a person points a tool at it. Developers paste a documentation site’s llms.txt or llms-full.txt into an AI coding assistant. That gives it accurate, current context. It is a real benefit, even if a narrow one.
llms.txt vs robots.txt (and Sitemaps)
The names are similar, which causes a lot of confusion. They do very different jobs.
| File | What it does | Who honours it | Can it block bots? |
|---|---|---|---|
robots.txt |
Tells crawlers which URLs they may or may not fetch | Formal standard, followed by all major search and AI crawlers | Yes, for crawlers that obey it |
sitemap.xml |
Lists the URLs you want found, with optional last-modified dates | Google, Bing and other search engines | No |
llms.txt |
Gives a hand-picked, readable summary and reading list for AI tools | No major search engine or AI search provider has committed to it | No |
The most important point: llms.txt cannot stop AI companies from crawling or training on your content. If that is your goal, you need robots.txt rules for user agents such as GPTBot, ClaudeBot and Google-Extended. Our guide to robots.txt and AI crawlers explains which tokens control what. I still see people confuse the two files, so this is worth saying twice.
In the same way, llms.txt does not replace your sitemap. Search engines find and refresh URLs through XML sitemaps and links. That is still true for AI features in Google Search.
Pros and Cons of Adding an llms.txt File
The Case for It
- It is cheap. A useful file for a small site takes under an hour to write, or one click to turn on in an SEO plugin.
- It cannot hurt Google rankings. Google has said directly that it neither helps nor harms visibility in Search.
- It helps developer audiences. If your customers use AI coding tools with your API or product, a clean llms.txt and Markdown docs help those tools give better answers about you.
- It makes you do a useful exercise. Deciding which ten to twenty pages best explain your business is good content strategy, whatever any bot does with the result. Honestly, this is the part I value most.
- It is ready if adoption changes. If an AI provider does start using it, you will not need to rush.
The Case Against It
- No proven visibility benefit. There is no public evidence from Google or the major AI providers that the file improves citations or traffic.
- Another file to look after. An out-of-date llms.txt pointing to old prices or deleted pages is worse than none, especially if a tool does read it.
- It can distract from work that matters. Crawlable pages, clear answers and a strong brand do more for AI visibility than any text file.
- Trust is a built-in weakness. Site owners write the file themselves. So an AI system has no reason to trust its claims over the pages it can crawl and check. That is the same problem that sank the keywords meta tag.
How to Create an llms.txt File
If you decide to publish one, do it properly. Here is the process I would follow.
- Pick the pages that really explain your site. Choose ten to thirty URLs: what you do, your main products or services, pricing, key guides, policies and contact details. Leave out tag pages, thin posts and anything you would not want quoted.
- Write a one-paragraph summary. Say plainly who you are, what you offer, who for and where. This goes in the blockquote. It is the part a model is most likely to rely on.
- Group links under clear H2 headings. Use headings such as “Services”, “Guides” or “Docs”. Add a short note after each link saying what the page answers.
- Put less important material under “Optional”. Archives, older posts and nice-to-know pages go here, so tools can skip them.
- Save it as UTF-8 and upload it to the root. The file should load at
https://yourdomain.com/llms.txt. It should return a 200 status code as plain text or Markdown, not an HTML page. - Check it and keep it current. Open it in a browser and confirm every link works. Add a review to your content calendar so it changes when your offer changes.
An Example llms.txt for a Service Business
Here is an example for a made-up accounting firm. The notes describe what each page answers, instead of repeating keywords.
# Northfield Accounting
> Northfield Accounting is a chartered accountancy firm in Leeds, UK,
> serving sole traders and small limited companies. We offer bookkeeping,
> year-end accounts, self-assessment tax returns and payroll.
Fixed monthly fees are listed on our pricing page. We do not offer
audit services or personal financial advice.
## Services
- [Bookkeeping](https://northfield.example/services/bookkeeping): Monthly bookkeeping using Xero or QuickBooks, what is included and turnaround times
- [Year-end accounts](https://northfield.example/services/year-end-accounts): Statutory accounts and corporation tax returns for limited companies
- [Self-assessment](https://northfield.example/services/self-assessment): Personal tax returns for sole traders and landlords, deadlines and documents needed
- [Payroll](https://northfield.example/services/payroll): Payroll for teams of 1 to 50, pensions auto-enrolment included
## Company information
- [Pricing](https://northfield.example/pricing): Fixed monthly packages and what each includes
- [About us](https://northfield.example/about): Our team, qualifications and professional body membership
- [Contact](https://northfield.example/contact): Office address, phone, opening hours and booking form
## Guides
- [Sole trader vs limited company](https://northfield.example/guides/sole-trader-vs-limited): Tax and admin differences explained
- [Self-assessment deadlines](https://northfield.example/guides/self-assessment-deadlines): Key HMRC dates and penalties
## Optional
- [Blog archive](https://northfield.example/blog): Older articles on small business tax
- [Careers](https://northfield.example/careers): Current vacanciesA documentation site would follow the same pattern, with sections such as “Getting started”, “API reference” and “Examples”. Ideally, it would link to the .md versions of each page.
Using an SEO Plugin Instead
On WordPress, Yoast SEO and Rank Math can create the file for you. That saves time, but review the output. Automatic files tend to list recent posts instead of your most important pages. The summary is also only as good as the site description you have entered. I’d edit the settings until the file reads like something you would hand to a new colleague.
Mistakes to Avoid
- Keyword stuffing the summary. Write so it is easy to understand. A model reading “best cheap accountant Leeds accountant near me” learns nothing useful.
- Listing every URL. That is a sitemap’s job. The value of llms.txt is in choosing.
- Linking to pages blocked in robots.txt or behind logins. A tool that follows the link will hit a wall.
- Saying things your pages do not. If the file claims you serve all of the UK but your pages say Yorkshire only, you create mixed signals.
- Expecting it to replace good pages. AI systems that cite you still read and quote your actual content.
Does Your Site Need llms.txt? An Honest Recommendation
Here is how I would decide, by type of site.
Software, SaaS and API Documentation: Yes
This is the use the proposal was designed for. Developers really do feed docs into AI assistants. A well-built llms.txt with Markdown pages, and maybe an llms-full.txt, makes your product easier to build with. If your docs platform creates it automatically, turn it on and check the output.
Publishers and Content-Heavy Blogs: Optional
A short file pointing to your main guides and editorial policies costs little. Do not expect a traffic change. Spend most of your effort on writing content AI assistants can quote, because that is what gets cited.
E-Commerce: Low Priority
Product catalogues change all the time, and a hand-written list goes stale fast. Your product feeds, structured data and crawlable category pages matter far more. If you add an llms.txt, keep it to policies, shipping, returns and top-level categories.
Local and Service Businesses: Fine to Skip
Nothing you lose by not having one can be measured today. If your plugin creates it with one click and you review it, there is no harm. Otherwise, your Google Business Profile, service pages and reviews are a better use of the time.
Large Enterprise Sites: Only With an Owner
A big site can publish one, but someone has to own it. Think about separate files for sections such as /docs/ or /support/, as the specification allows, instead of one huge root file.
How to Tell Whether It Is Doing Anything
Do not guess. Check your server logs for requests to /llms.txt. Note which user agents fetch it and how often. Our beginner’s guide to log file analysis shows how. Then compare that with referral traffic from AI assistants in GA4. If nothing fetches the file and nothing changes, you have your answer.
Conclusion: Treat llms.txt as a Low-Cost Extra, Not a Strategy
llms.txt is a sensible idea that few have adopted. As of late September 2026, Google Search ignores it, and no major AI search provider has said it relies on it. It does help where people choose to point AI tools at documentation.
If you run a docs or developer site, publish one and keep it accurate. For everyone else, it is a quick optional task. In my view, it should never jump ahead of crawlable pages, clear answers and solid technical SEO. A good next step is to see whether AI assistants already send you visitors. Set up AI referral tracking in GA4 so you can measure what actually moves.
