Understanding LLMs.txt: controlling AI search visibility in the age of LLMs

We already know how to measure the effectiveness of AI Search Optimization, but how can we make our content appear more frequently in generative chat interfaces? One emerging tool is the LLMs.txt file – a new proposal designed to bridge the gap between websites and large language models (LLMs) like ChatGPT, Gemini, or Claude.

Key takeaways

  • llms.txt is a proposed Markdown file at yourdomain.com/llms.txt that lists a site’s most important pages, with short descriptions, for large language models.
  • It’s not a standard, and no major AI search platform has confirmed using it. Google says it doesn’t support llms.txt and that no special AI files are needed.
  • The data shows little effect: in an Ahrefs study of 137,000 domains, 97% of llms.txt files received no requests at all in May 2026, and a Trakkr analysis found no difference in AI citations.
  • It doesn’t control access. Use robots.txt to allow or block AI crawlers.
  • When it’s worth having: documentation and developer sites used by AI agents, or as a low-cost addition once the fundamentals (crawlable HTML, clear content, brand mentions) are in place.

As AI assistants such as ChatGPT, Gemini, Claude and Perplexity answer more questions, website owners look for ways to influence what these systems read. llms.txt is one of the most discussed ideas: a file that tells language models which pages matter most. It’s easy to create, and many SEO plugins now generate it automatically.

But does it work? This article explains what llms.txt is, how to write one correctly, what studies and Google say about its effect, and when it’s worth the effort.

Let your customers find you, wherever they look.

What is llms.txt?

llms.txt is a proposal published in September 2024 by Jeremy Howard of Answer.AI. It’s a Markdown file placed in the root of a website that gives language models a short, structured overview: what the site is, and links to its most useful pages with one-line descriptions. The idea is that a model can read one clean file instead of crawling complex HTML.

It’s still a proposal, not a web standard, and no organisation governs it.

robots.txtsitemap.xmlllms.txt
PurposeAllow or block crawlingList pages for indexingSummarise and link key content for LLMs
FormatPlain-text rulesXMLMarkdown
StatusInternet standard (RFC 9309)Standard supported by search enginesProposal
Controls AI access?Yes, per user agentNoNo
Confirmed use by AI searchYes (crawlers respect it)Yes (search engines)Not confirmed

Does llms.txt improve AI search visibility?

So far, the evidence says no:

  • Most files are never read. Ahrefs analysed server logs for 137,000 domains: 28% published an llms.txt file, and 97% of those files received zero requests in May 2026. Most requests that did arrive came from SEO audit tools and other bots, not AI platforms, and AI bots never requested llms.txt on sites that didn’t have one.
  • AI crawlers rarely fetch it. In a 12-week test on 83 sites, GPTBot fetched llms.txt 7 times against 3,990 robots.txt fetches, ClaudeBot 9 times against 3,120, and PerplexityBot not at all.
  • No difference in citations. Trakkr compared 37,894 AI-cited sites: those with llms.txt averaged 6.83 AI citations, those without 6.73.
  • Google doesn’t use it. Gary Illyes said Google doesn’t support llms.txt and isn’t planning to, John Mueller compared it to the old keywords meta tag, and Google’s AI search guide says you don’t need special files or markup for AI features.

This doesn’t mean the idea is useless; it means llms.txt isn’t a lever for appearing in AI answers today. What AI search uses is the same as search: crawlable, indexed pages, clear content and a brand that’s mentioned elsewhere. See how AI chatbots choose what to recommend and 10 AI search myths.

Doesn’t llms.txt help with JavaScript-heavy sites?

Not in practice. It’s true that major AI crawlers don’t render JavaScript, so content loaded by scripts can be invisible to them. But a crawler that doesn’t fetch llms.txt can’t use it as a shortcut. The fix is to serve key content in the HTML the server sends.

When is llms.txt worth having?

  • Developer documentation and APIs. Coding assistants and AI agents increasingly read documentation; llms.txt (and llms-full.txt) gives them a clean entry point. Chrome Lighthouse has added an llms.txt check as part of agent readiness.
  • Sites where AI agents need canonical facts: policies, pricing pages, product catalogs and support content an agent might look up on a user’s behalf.
  • As a low-cost extra. Creating a good file takes an hour or two. If your fundamentals are in place, there’s little downside; just don’t expect it to change your visibility.

It’s not worth prioritising over: crawlable HTML, clear and current content, Bing and Google indexing, brand mentions and accurate information and measurement.

Get found by AI. Grow beyond Google.

How do you create an llms.txt file?

  1. Start with an H1 containing the name of your site or company (the only required element).
  2. Add a short summary in a blockquote: what you do, for whom, where.
  3. Optionally add a paragraph with key facts or notes for models.
  4. Add H2 sections with Markdown lists of links, each followed by a colon and a one-line description (see the example below).
  5. Use an “Optional” section for secondary links that can be skipped.
  6. Place it at yourdomain.com/llms.txt, serve it as plain text, and keep it updated with your site.
  7. Control access separately in robots.txt, for example by allowing OAI-SearchBot so your pages can appear in ChatGPT search.

Example for an online store, following the llmstxt.org format:

# Example Store

> Example Store is an online shop for natural cosmetics in Poland,
> with 1,200 products, delivery in 24 hours and free returns within 30 days.

## Products
- [Face care](https://example.com/face-care/): serums, creams and cleansers by skin type
- [Vitamin C serums](https://example.com/vitamin-c-serums/): comparison of 12 serums, prices and ingredients

## Customer information
- [Delivery and returns](https://example.com/delivery-returns/): delivery times, costs and return policy
- [About us](https://example.com/about/): company details, certifications and contact

## Optional
- [Blog](https://example.com/blog/): skincare guides and ingredient explainers

How do you check whether it’s being used?

Check your server logs for requests to /llms.txt and note the user agents (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and others). Compare with how often the same bots fetch robots.txt. For visibility, measure AI citations and mentions before and after, as described in metrics that matter in AI search. Our observations after implementing llms.txt for clients are in llms.txt insights after implementation.

Summary

llms.txt is a sensible idea with little adoption on the AI side so far. It doesn’t control access, isn’t used by Google and hasn’t been shown to increase AI citations. Treat it as an optional, low-cost file for documentation and agent use cases, and put your effort where AI search visibility actually comes from: accessible content, strong SEO and a well-described brand. See also how to rank in ChatGPT.

Boost sales with SEO + GEO.

FAQ: llms.txt

What is llms.txt?

A proposed Markdown file at the root of a website that summarises the site and links to its most important pages for large language models. It was proposed by Jeremy Howard in September 2024.

Does llms.txt help SEO or AI search visibility?

There’s no evidence that it does. Google doesn’t use it, Ahrefs found 97% of llms.txt files received no requests, and a Trakkr analysis found no difference in AI citations between sites with and without it.

Does Google use llms.txt?

No. Google’s Gary Illyes said Google doesn’t support it and isn’t planning to, and Google’s AI search guide says no special AI files are needed.

What’s the difference between llms.txt and robots.txt?

robots.txt is a standard that tells crawlers what they may crawl; AI crawlers respect it. llms.txt only lists recommended content and can’t block anything.

Should I create an llms.txt file?

If you run developer documentation or want AI agents to find canonical information, yes. For most marketing sites it’s optional: harmless and cheap, but not a priority.

What is llms-full.txt?

A companion file some sites publish with the full text of their key pages in Markdown, mainly used for documentation that coding assistants load as context.

Sources

About the author: Joanna Jelenik is Delivery Manager at Insightland. Insightland is a Wrocław-based SEO and GEO agency, part of the Hexe Capital Tech Hub. Contact us.

more

Related blog posts

AI Search Content

Using AI Search insights to shape your content strategy

16 Oct 2025 • Insightland

two-people-talking-about-seo
Content Monthly updates SEO Technical SEO

From success to insights – January 2025

21 Feb 2025 • Insightland

E-commerce

SEO in an online shop – how to cost-effectively generate profits using free traffic

21 Jul 2021 • Insightland