How to Write a Good llms.txt File for Your Site

AI assistants read your site differently than people do. They want clean, structured context, and they want it fast. An llms.txt file gives it to them. This guide shows you how to write one that is clear, curated, and easy to maintain. You will learn what to include, what to skip, and how to test the result. Along the way, we keep expectations honest, so you invest effort where it pays off.

Updated: October 05, 2026

10 min read

Share:

Share via Linkedin

Share via X

Share via Facebook

Copy page link

What an llms.txt File Does

An llms.txt file is a Markdown document that sits at the root of your domain. Visitors rarely see it. AI tools and agents can fetch it to understand your site quickly. Think of it as a reading list for machines, written by the people who know the content best.

Jeremy Howard of fast.ai proposed the format in September 2024. The problem was simple. Language models have limited context windows, and modern web pages are cluttered with navigation, scripts, and styling. A short, curated summary points a model straight to the pages that matter.

It differs from the files you already know. A sitemap lists every indexable URL for search engines. A robots.txt file sets crawl permissions. An llms.txt file does neither. It explains your site and recommends where to start. Our session on the future of search explores how the shift toward AI-agent optimization is reshaping content strategy.

Set Realistic Expectations Before You Start 🎯

Here is the honest picture. The format is an emerging convention, not a ratified standard. No major standards body has adopted it. Google has said the file will not help or hurt your Search rankings. Do not expect a traffic spike.

Evidence of real-world use is mixed. A June 2026 Ahrefs study of server logs from 137,000 domains found that 97% of llms.txt files received zero requests in May 2026. Adoption is still modest, too. A June 2026 tracker found that 8.7% of the top 1,000 websites publish one.

So why bother? The cost is low, and the upside is real for some audiences. Coding assistants and documentation tools consume these files directly. Chrome’s Lighthouse tooling now includes an optional audit for the file. Treat it as a small, sensible investment, not a growth hack.

Know the Anatomy of the File

The format is deliberately minimal. Every file follows the same pattern. Start with a single H1 heading that names your site or project. That is the only required element. Everything else is optional, but a good file uses most of it.

Next comes a blockquote with a short summary. Add a few lines of plain context if needed. Then add H2 sections, each holding a Markdown list of links. Every list item uses a link, followed by a colon and a short note.

One H2 section has special meaning: Optional. Links placed there can be skipped when a tool needs a shorter context. The tabs below walk through each part with a working example.

Write a Clear Title and Summary

Your H1 should be your brand or site name. Skip slogans. Models use it to label everything that follows, so clarity beats cleverness.

The summary matters most. Write one to three sentences. State who you are, what you offer, and who you serve. Use plain language. Avoid superlatives and sales copy, because a model gains nothing from them.

Test your summary with a simple check. Could a stranger repeat it after one read? If not, simplify. Add key facts below the summary only when they prevent misreadings, such as product names or regions served.

Choose the Pages That Matter Most

Curate ruthlessly. The most common mistake is treating the file like a sitemap. A model gains nothing from two thousand links. It needs the twenty that explain your business.

Start with your core pages. Include product or service overviews, pricing, documentation, policies, and your best explainer content. Prefer pages with durable, factual information. Skip thin pages, tag archives, and login screens.

A useful rule of thumb is ten to thirty links for most sites. Large documentation sites may need more. Ask one question of each candidate. If a model could only read this page, would it help answer a customer question?

Organize Links Into Logical Sections

Group links under H2 headings that mirror how people think about your business. Common choices include Products, Documentation, Pricing, Resources, and Company. Keep section names short and literal.

Order matters. Put the most important section first, and the most important link first within it. Tools often read from the top. Place your strongest material where it is seen early.

Aim for three to six sections. Fewer than three can look thin. More than six starts to resemble a sitemap. Consistent naming helps too. Use the same pattern for every section and every link.

Write for Machines, Edit for Humans

Treat the file like documentation. A teammate should be able to read it in two minutes and understand your site. If a human finds it clear, a model usually will too.

Write Link Descriptions That Earn Their Place

The note after each link is your best tool. A bare link tells a model little. A good description explains what the page contains and when it is useful. One sentence is plenty.

Be specific. Compare “Pricing page” with “Pricing for all plans, billing terms, and usage limits.” The second tells a model exactly when to fetch the page. Lead with the content type, then add the key detail.

Stay factual. Skip adjectives like “powerful” or “innovative.” State what the page covers, who it is for, and what it answers. Accurate descriptions build trust with every tool that reads your file.

Use the Optional Section Wisely

The Optional section is a pressure valve. Links there signal lower priority. A tool with limited context can drop them and keep everything else. Use it for material that is helpful but not essential.

Good candidates include older documentation versions, deep reference pages, press coverage, and extended case studies. Keep policies and core product pages in main sections. Anything a model must know belongs above the Optional heading.

Do not use it as a dumping ground. If a page is neither essential nor useful, remove it entirely. A shorter file is easier to maintain and easier to trust.

Consider a Companion llms-full.txt File

Many sites also publish llms-full.txt. It is a convention that sits alongside the original proposal. Instead of linking to pages, it holds their full text in one Markdown file.

This helps tools that want deep context in a single request. Documentation sites use it most. A June 2026 tracker found that 15 of 87 top sites with an llms.txt file also publish a full version.

Start with llms.txt alone. Add the full file only if you have clean Markdown versions of your key pages. A bloated full file can overwhelm the very context window you are trying to protect.

Publish, Test, and Validate Your File

Save the file as plain UTF-8 text named llms.txt. Upload it to your domain root, next to robots.txt. The final address should look like yoursite.com/llms.txt. Many content platforms and plugins can generate the file, but always review the output.

Then test it. Load the URL in a browser and confirm it returns a successful response. Check that the content type is plain text or Markdown. Click every link. Broken or redirected links undermine the file’s purpose.

Finally, try it with an AI assistant. Paste the file into a chat and ask questions about your business. Do the answers point to the right pages? If not, refine your summary and descriptions.

Test Before You Trust

Run a quick check before you publish. Open the file, click every link, and ask an AI assistant a few questions about your business. Five minutes of testing catches most problems.

Keep the File Fresh Over Time ⚡

An outdated file is worse than none. It points tools to stale pages and wrong facts. Schedule a review each quarter, and after any major launch, rebrand, or restructure.

Tie updates to your publishing workflow. When a key page moves, update the link the same day. When you retire a product, remove it. Assign a clear owner so the task does not slip.

Monitor your server logs if you can. Requests to /llms.txt show whether any tools are fetching it. Given current evidence, expect modest numbers. Use the data to decide how much effort the file deserves.

Common Mistakes to Avoid

Four mistakes show up again and again. First, dumping every URL into the file. Second, writing promotional copy instead of plain descriptions. Third, linking to pages that redirect or return errors. Fourth, never updating it after launch.

Another trap is confusing llms.txt with access control. It does not block or permit crawlers. Use robots.txt for that job. Never list private or sensitive pages, since the file is public by design.

Last, avoid overpromising. Do not tell stakeholders the file will lift rankings or guarantee AI citations. Position it as a low-cost clarity measure. That framing keeps everyone aligned, and it holds up under scrutiny.