Every few months a new “must-have” file for AI search appears in SEO newsletters. In 2025 and 2026 that file has been llms.txt: a short Markdown document at the root of your site that promises to tell ChatGPT, Claude, Gemini and Perplexity what your website is about and which pages matter most.
The idea is sensible. The evidence that it affects AI search visibility is, so far, very thin. Google has said plainly that it doesn't need such files, and no major AI search platform has confirmed using one. That doesn't make llms.txt useless, but it does change how much time it deserves.
This guide explains what llms.txt is, what the format looks like, what the platforms have actually said, who might still benefit, and how to create one in about 30 minutes if you decide it is worth it.
Key Takeaways
- llms.txt is a proposal, not an adopted web standard. It was published by Jeremy Howard on llmstxt.org in September 2024.
- The file is Markdown: a site name, a short summary and curated lists of links, optionally pointing to clean
.mdversions of pages. - Google says you don't need AI text files to appear in AI Overviews or AI Mode, and Google's John Mueller has compared llms.txt to the keywords meta tag.
- No major AI search engine has publicly confirmed using llms.txt for crawling, ranking or citations.
- It is most useful for documentation-heavy and developer-focused sites; for most businesses it is optional and low priority.
What is llms.txt?
llms.txt is a plain-text file written in Markdown and placed at /llms.txt on your domain (or under a subpath). The proposal on llmstxt.org summarises its purpose as “a proposal to standardise on using an /llms.txt file to provide information to help agents use a website.”
The problem it tries to solve is real. Web pages are full of navigation, scripts, cookie banners and ads. Language models work best with clean, concise text and have limited context windows. A curated file that says “here is what this site is, and here are the ten pages that explain it best” could, in theory, help an AI agent find the right information faster.
llms.txt vs robots.txt vs sitemap.xml
| File | Purpose | Format | Status |
|---|---|---|---|
robots.txt | Tells crawlers which URLs they may fetch | Plain-text directives | Long-established, formalised as RFC 9309, honoured by major search crawlers |
sitemap.xml | Lists URLs you want discovered and crawled | XML | Supported by Google, Bing and others |
llms.txt | Curated summary and links for AI agents | Markdown | Community proposal; no confirmed use by major AI search engines |
The key difference: robots.txt controls access, and an XML sitemap aids discovery. llms.txt does neither. It cannot block AI crawlers or make them visit you. If you want to control access, our guide to robots.txt vs meta robots is the place to start.
The llms.txt format
According to the specification, sections must appear in this order:
- An H1 with the name of the project or site. This is the only required section.
- A blockquote with a short summary containing key information.
- Optional paragraphs or lists with more detail (no headings).
- Zero or more H2 sections, each containing a list of Markdown links with optional notes.
- An “Optional” H2 section, by convention, for secondary links an agent can skip if it is short on context.
Here is a simple example for a local service business:
# Example Plumbing Co.
> Family-run plumbing company serving Austin, Texas since 2009.
> Emergency repairs, water heaters, drain cleaning and repiping.
Licensed and insured. Prices are quoted on site; call-out fee applies.
## Services
- [Emergency plumbing](https://example.com/emergency/): 24/7 response area and what to do before we arrive
- [Water heater installation](https://example.com/water-heaters/): tank vs tankless, typical timelines
## Help
- [Pricing guide](https://example.com/pricing/): how our quotes are calculated
- [Service area](https://example.com/areas/): ZIP codes we cover
## Optional
- [About us](https://example.com/about/)
The proposal also suggests offering a clean Markdown version of key pages at the same URL with .md appended (for example /pricing.md), and some tools generate a larger companion file, often called llms-full.txt, containing the full text of many pages.
Who actually uses llms.txt? What the platforms say
This is the part most “how to” articles skip. Here is what has been said publicly, as of October 2026.
Google's official documentation on AI features is explicit:
“You don't need to create new machine readable files, AI text files, or markup to appear in these features.”
— Google Search Central, AI features and your website
In April 2025, Google's John Mueller wrote on Reddit that, to his knowledge, none of the AI services had said they were using llms.txt, and that “to me, it's comparable to the keywords meta tag,” noting that server logs showed bots weren't even checking for it (reported by Search Engine Journal). In July 2025, attendees at a Google Search Central event reported Gary Illyes saying Google doesn't support llms.txt and isn't planning to; that report is second-hand, but it is consistent with Google's documentation.
AI search and chat platforms
OpenAI, Perplexity and Microsoft document their crawlers and robots.txt behaviour, but none has publicly confirmed using llms.txt to choose sources for answers. Perplexity's crawler documentation, for example, talks only about robots.txt for PerplexityBot. Some AI companies, including Anthropic, publish llms.txt files for their own documentation, which shows interest in the format, not that their crawlers rely on it.
What tests show
Semrush reported in its own llms.txt analysis that a test file on one of its sites received no visits from GPTBot, PerplexityBot, ClaudeBot or Google-Extended between mid-August and late October 2025, and that it saw no correlation between adding the file and improved AI visibility. The same article cites NerdyData figures showing adoption growing to about 16,670 domains by September 2026: rapid growth, but a tiny fraction of the web.
| Platform | Public position on llms.txt (Oct 2026) |
|---|---|
| Google Search (AI Overviews, AI Mode) | Not needed; Google says no AI text files are required |
| OpenAI (ChatGPT search) | No public confirmation of use; access is governed by OAI-SearchBot and robots.txt |
| Perplexity | No public confirmation of use; PerplexityBot follows robots.txt |
| Microsoft (Bing, Copilot) | No public confirmation of use |
| Coding assistants and agent frameworks | Some tools can read llms.txt when a user or developer points them to it |
Should you use llms.txt? A decision guide
Use the questions below rather than following the hype either way.
- Do you publish technical documentation, an API or a developer product? Then yes, it is worth a try. Developers increasingly paste documentation links into coding assistants, and a clean Markdown index is genuinely convenient.
- Is your SEO foundation solid? If you still have indexing errors, slow pages or thin content, fix those first. They affect every search and AI platform. Our technical SEO guide lists the priorities.
- Can you maintain it? An outdated llms.txt that links to discontinued products is worse than none.
- Are you hoping it will get you into AI answers? Then set expectations: there is no evidence it does. Work on generative engine optimization, brand mentions and crawl access instead.
How to create an llms.txt file in six steps
- Pick 10–30 pages. Choose evergreen, high-value pages: core services or products, pricing, key guides, policies and contact details.
- Write the summary. One or two factual sentences: who you are, what you do, where you operate.
- Group links under H2 headings. Use plain labels such as Services, Guides, Policies. Add a short note after each link.
- Save as UTF-8 Markdown named
llms.txtand upload it to your site root so it loads athttps://yourdomain.com/llms.txt. - Check it loads as plain text with a 200 status code, and that every link works.
- Add it to your content calendar. Review it whenever you launch, retire or significantly change key pages.
Some WordPress SEO plugins and site builders can generate the file automatically. If you use one, review the output: automatic files often list every post rather than a curated selection, which defeats the purpose.
Common mistakes with llms.txt
- Treating it as a ranking factor. It isn't one for Google, and no AI search engine has said otherwise.
- Using it to “block” AI. llms.txt has no access-control function. Use robots.txt and, if needed, server or CDN rules.
- Stuffing it with keywords or marketing copy. If an agent does read it, it wants facts and clean links, not slogans.
- Listing every URL. That's what your sitemap is for. Curate.
- Forgetting the basics. Accidentally blocking OAI-SearchBot or PerplexityBot in robots.txt will hurt your AI visibility far more than a missing llms.txt.
What to do instead (or as well)
If your goal is to appear in AI answers, these actions have much clearer support:
- Allow the search crawlers you want to be cited by (Googlebot, Bingbot, OAI-SearchBot, PerplexityBot).
- Make important content render as text in HTML. See our JavaScript SEO guide.
- Answer specific questions clearly near the top of pages, with evidence.
- Earn mentions on trusted third-party sites.
- Measure AI referrals and citations; our guide to tracking brand visibility in AI search explains how.
For platform-specific tactics, see our guides to Perplexity SEO, Google AI Mode and ChatGPT search optimization.
Related Guides
- How to Track Your Brand’s Visibility in AI Search
- AI SEO Tools: What They Do Well (and What They Don’t)
- ChatGPT and Claude Prompts for SEO: 40 Templates
Frequently Asked Questions
What is llms.txt?
llms.txt is a proposed standard, published by Jeremy Howard in September 2024, for a Markdown file at /llms.txt that gives large language models and AI agents a short summary of a website and a curated list of links to its most useful content.
Does Google use llms.txt?
No. Google's AI features documentation says you don't need to create new machine-readable files or AI text files to appear in AI Overviews or AI Mode, and Google's John Mueller has compared llms.txt to the keywords meta tag. Google Search does not use it as a ranking or eligibility signal.
Does ChatGPT or Perplexity read llms.txt?
None of the major AI search platforms has publicly confirmed that its crawlers use llms.txt for search or citations. Semrush reported that its own test file received no visits from GPTBot, PerplexityBot or ClaudeBot over roughly two months in 2025.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers what they may or may not fetch and is widely honoured. llms.txt is an optional, curated guide to content and does not grant or block access to anything.
Can llms.txt hurt my SEO?
Not directly. It is a plain file that Google ignores for ranking. The real risks are wasting time that could go into proven SEO work, and letting the file drift out of date or link to pages you no longer want promoted.
Who should create an llms.txt file?
Sites with technical documentation, APIs or developer products are the best fit, because coding assistants and agents may be pointed at the file directly. For most small business websites it is optional and low priority.
Conclusion
llms.txt is a tidy idea with an uncertain future. It costs little to create, and it may help developer tools and future agents, but today it is not a lever for Google rankings or AI search citations. Treat it as an optional extra once your crawl access, content quality and brand authority are in good shape. If you want a clear, evidence-based plan for AI search visibility, our SEO team can audit your site and prioritise what actually moves results. Request a free quote to get started.
References
- The /llms.txt file – llmstxt.org
- AI features and your website – Google Search Central
- Google Says LLMs.Txt Comparable To Keywords Meta Tag – Search Engine Journal
- Google says normal SEO works for ranking in AI Overviews and llms.txt won't be used – Search Engine Land
- What is llms.txt & should you use it? – Semrush Blog
- Perplexity Crawlers – Perplexity documentation
