llms.txt: what it is for and when to add it

Understand the difference between Google Search requirements and optional reference files for AI agents.

A librarian uses a small directory map to point towards source pages while a visitor reads one of the pages.

Somebody on LinkedIn has told you your website needs an llms.txt file, and that without one AI assistants will wander past your business like it's a closed-down petrol station. It's an easy job to sell: one file, sounds technical, done by lunch. The honest answer is that it depends which AI system you're asking about, and what you want it to do. For Google Search, Google has already answered. For AI agents and coding tools, it's a different question.

What llms.txt is

llms.txt is a proposal, not an official web standard. It's a markdown file that gives AI agents a short guide to a website: what the site is, and links to the pages worth reading.

The current version of the proposal, v2, says the file can sit at the root of a site or at any path within it, covering the pages under that path. Where more than one applies, agents should use the most specific. So a business could have one at /llms.txt, and a software company could have another at /docs/llms.txt just for its documentation.

It's often compared with robots.txt, but they do different jobs. robots.txt gives crawlers instructions about where they may go. Those instructions are voluntary, and they don't replace real access controls such as a login or a firewall. llms.txt offers context, guidance and links for systems that choose to read it; it does not enforce access restrictions.

What Google Search says

Google's guide to its generative AI features in Search is clear. You don't need new machine-readable files, AI text files or markdown to appear in Google Search or its AI features, because Google Search doesn't use them. Keeping an llms.txt file won't help or harm your visibility in Google Search. Google also says it's fine to keep one for other services that do use these files.

That answer is about Google Search. It isn't a statement about every AI product, including Google's own.

Where llms.txt is actually used

The places it does turn up:

  • AI agents and coding tools. The v2 proposal says documentation platforms generate the file automatically, and that the AI labs publish llms.txt files for their own developer docs.
  • Chrome's Lighthouse. Lighthouse has an "agentic browsing" category that includes an llms.txt audit. Chrome's documentation says that if your site doesn't serve the file, the audit is marked Not Applicable, because providing it is optional for now. It flags a server error when fetching the file.

So Google's search team and Google's Chrome team say different things about the same file. They're answering different questions: appearing in search versus being usable by an agent browsing on someone's behalf.

Is it worth it for your site?

Don't decide on a hunch. Ask:

  1. Which system do I want it for? Name the actual product: a search service, a coding assistant or a browsing agent.
  2. Does that system document llms.txt support? Check its own documentation. If it doesn't mention it, don't assume.
  3. Do I have content an agent needs to navigate? A documentation library gives you a concrete use case to test; a small brochure site may not need an additional guide.
  4. Can I keep it accurate? A summary that drifts away from your real pages can mislead anyone relying on it.
  5. Can I see whether it's used? Check your server logs for requests to the file after you publish it. A request shows a fetch, not that the contents influenced an answer or recommendation.

If your platform already generates one, check it describes your site accurately and leave it in place.

What to fix first either way

Whatever you decide about llms.txt, the pages themselves still do the heavy lifting:

  1. Check access. Make sure the crawlers you care about aren't blocked in robots.txt, and that your important pages can be indexed. Our guide to checking whether ChatGPT can find your website explains how to separate search access from training controls.
  2. Put the words in the page. If your service descriptions only appear after scripts run, or are baked into images, some systems may not read them.
  3. Give each page a clear title and description, to give search services a useful starting point; they may choose different wording.
  4. Keep business details consistent across your pages and listings. Our WordPress guide to business details and schema covers the WordPress side.
  5. Answer real customer questions on the page, in plain language, with evidence a reader can check.

Keep the evidence with the changes

Whatever you change, record what you checked, what you changed and what the next check showed. CAFMO is built around that loop:

  • It checks your pages and explains each finding at Beginner, Novice or Developer level.
  • It lets you recheck a page after you publish a change, and saves the fresh result with its date.

See how CAFMO checks work.

Frequently asked questions

Will adding llms.txt hurt my website?

Google says it won't help or harm visibility in Google Search. For other systems, check their own documentation. The main practical risk is a summary that no longer matches your pages.

My platform created an llms.txt automatically. Should I delete it?

There's no need. Check it describes your site accurately and leave it.

Is llms.txt the same as robots.txt?

No. robots.txt gives crawlers voluntary instructions about where they may go. llms.txt offers a summary and links for systems that choose to read it. Neither one is a lock: if something must stay private, put it behind a login.

Sources and further reading

Create your CAFMO accountSee how CAFMO worksCompare plans