What happened after we implemented llms.txt?
The LLMs.txt file is not yet an industry standard, but its implementation on a website is recommended as part of the AI Search Optimization (AISO) process. Here's a brief summary of the results we’ve observed after implementing LLMs.txt.
Key takeaways
- We added an llms.txt file on 18 February 2025 and tracked AI bot visits to the site.
- GPTBot visits jumped the next day (from a few hundred to almost 2,000), and Perplexity-User fetched our content several times. We found no proof that GPTBot read the llms.txt file itself, so we can’t say the file caused the jump.
- Larger studies since then point the same way: 97% of llms.txt files get no requests (Ahrefs), there’s no difference in AI citations (Trakkr), and Google doesn’t use it (SEL).
- Our view: llms.txt is cheap and harmless, but it’s not an AI visibility strategy. Content in plain HTML, structure and brand mentions matter more.
We wrote about llms.txt in theory. Then we tested it and looked at how AI bots behaved on the site. Here’s what we saw, how to read it, and what we’d recommend today.
How did we measure it?
| Site | The website on which Insightland implemented the file (homepage, articles and subpages) |
| Implementation date | 18 February 2025 |
| Data source | Server logs: requests by AI bot user agents |
| Period analysed | The days before and after implementation (18–21 February 2025) |
What did we observe?
| Observation | Data | How to read it |
|---|---|---|
| GPTBot visits | From a few hundred (18 Feb) to almost 2,000 (19 Feb) | GPTBot crawls for training in waves; one day isn’t enough to link it to llms.txt, and we found no GPTBot request for the file itself |
| Bots requesting llms.txt | Bing, Amazon, TikTok/ByteDance, Moz, PetalBot | Mostly search, SEO and other crawlers; requesting a file doesn’t mean using it |
| Perplexity-User | 7 requests in 3 days, for pages and images | Fetched in response to users’ questions, so our content was likely used in answers; unrelated to llms.txt |
| Organic rankings | No negative change | llms.txt doesn’t affect classic SEO |
Make your brand visible – from Google to AI chatbots.
What doesn’t our data prove?
- Causation. A spike after a change isn’t proof the change caused it. GPTBot is OpenAI’s training crawler (OpenAI) and its activity varies over time on many sites.
- That the file was read by AI platforms. We found no direct evidence that GPTBot or OAI-SearchBot requested llms.txt.
- More citations. Bot visits are not citations. To measure AI visibility you need data on answers, not crawls.
What do larger studies show?
| Study | Finding |
|---|---|
| Ahrefs, 137,000 domains, 2026 | 28% of domains had llms.txt; 97% of those files received no requests in May 2026; AI bots didn’t request llms.txt on sites without one |
| Trakkr, 37,900 cited domains, 2026 | Pages with llms.txt averaged 6.83 AI citations, without it 6.73: no meaningful difference |
| Google doesn’t use llms.txt; normal SEO is what matters for AI Overviews |
What is llms.txt and how does it work?
llms.txt is a proposed standard: a Markdown file in the site’s root (yourdomain.com/llms.txt) that lists key pages with short descriptions, so language models can find important content quickly (llmstxt.org). It can highlight key pages and give context. It can’t block content: to keep AI bots away from parts of a site, use robots.txt rules for each bot (Google). And it doesn’t replace content in HTML: most AI crawlers don’t run JavaScript (Vercel), so the fix for JavaScript-heavy sites is server-side rendering, not a separate file.
Is llms.txt worth implementing?
| Worth it | Don’t expect |
|---|---|
| Low cost, no SEO risk | More AI citations or traffic |
| A useful map for developer documentation and AI agents that look for it | Google using it |
| Forces you to define your key pages and messages | A substitute for content in HTML, structure or brand mentions |
What moves AI visibility more: content that answers questions clearly and is visible in plain HTML, a sound technical foundation, and mentions of your brand on trusted sites (how AI recommendations work, AI search myths).
How can you check whether AI bots read your llms.txt?
Short answer: look at your server logs, not at analytics. Requests from AI bots don’t run JavaScript, so tools such as GA4 don’t record them.
- Filter your server or CDN logs for requests to /llms.txt and to the pages it lists.
- Group requests by user agent: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot and Perplexity-User (Perplexity), ClaudeBot (Anthropic) and others. Each has a different purpose: training, building a search index, or fetching a page for a user’s question.
- Verify that the bots are real. User agents can be faked; OpenAI and Perplexity publish the IP ranges of their crawlers, so check requests against them.
- Compare periods. Look at the same days of the week before and after adding the file, and note other changes made in that time (new content, sitemap updates), so you don’t credit the file for their effect.
- Measure citations separately. A crawl isn’t a citation. Track a fixed set of prompts in AI assistants and see whether your pages are cited (metrics that matter).
| What you see in the logs | What it means |
|---|---|
| GPTBot requests | Crawling for model training |
| OAI-SearchBot, PerplexityBot requests | Crawling for the platform’s search index |
| ChatGPT-User, Perplexity-User requests | A page fetched to answer a user’s question; the closest signal of possible use in an answer |
| A request to /llms.txt | The file was downloaded; this doesn’t show that it was used |
What are the next steps?
A correct llms.txt file is just one item on the checklist for optimizing brand visibility in AI-generated answers. AI Search Optimization also needs a new approach to measuring key metrics and to structuring website content (metrics that matter).
More in our AI Search series:
- AI Search 101: what business owners need to know >>
- Behind the algorithms: how AI chatbot recommendations work >>
- How to rank in ChatGPT? [2025] >>
- Measuring success in AI Search: metrics that matter >>
Get Found by AI. Grow Beyond Google.
AI Search is transforming how customers discover brands. Optimize your presence in AI-driven answers and start capturing high-intent traffic today.
FAQ
Does llms.txt increase AI citations?
There’s no evidence that it does. Our own data showed more bot visits, but no proof the file was used; larger studies found no difference in citations.
Does Google use llms.txt?
No. Google has said it doesn’t use llms.txt for AI Overviews or Search.
Can llms.txt block AI bots?
No. Use robots.txt rules for each AI user agent.
Should I remove llms.txt?
No need. It does no harm; just don’t rely on it for AI visibility.
Sources
- llmstxt.org: The /llms.txt file
- Ahrefs: We analyzed 137K sites – 97% of llms.txt files never get read
- Trakkr: AI crawler and citation data
- Search Engine Land: Google says llms.txt won’t be used
- OpenAI: Overview of OpenAI crawlers
- Perplexity: Perplexity crawlers
About the author: Joanna Jelenik is Delivery Manager at Insightland. Insightland is a Wrocław-based SEO and GEO agency, part of the Hexe Capital Tech Hub. Contact us.
