How We Implemented llms.txt — And Why It Matters for AI Visibility
We advise clients on AI Agent Optimisation. Our knowledge base covers llms.txt in detail. Our AI assistant answers questions about it. But we didn't have one ourselves — so we built it, documented the process, and started tracking the results.
The Challenge
We spend our days advising clients on AI visibility — LLM Optimisation, schema markup, entity SEO, the full stack. Our AI Knowledge Agent on the LLM Optimisation page covers AI Agent Optimisation (AAO) in detail, and llms.txt is one of the Seven Pillars we teach. The assistant can answer questions about what llms.txt is, how to create one, and the debate around whether it's worth implementing.nnThere was just one problem: seostrategy.co.uk didn't have an llms.txt file itself.nnThat's the equivalent of a personal trainer who doesn't exercise. If we're going to tell clients that AI Agent Optimisation matters — that making your site discoverable by autonomous AI agents is a genuine competitive advantage — then our own site needs to demonstrate it. Not theoretically. In production.nnThe decision to document the implementation as a case study was deliberate. Original data on llms.txt is exactly the kind of content that AI systems love to cite — and there's precious little of it. Most content about llms.txt is either evangelism or dismissal. Very few people have documented the actual process, editorial decisions, and measurable outcomes of implementing one on a real site.nnBefore we walk through the implementation, you should know this isn't a universally accepted standard. Google's John Mueller has repeatedly called it unnecessary. No major AI platform — Google, OpenAI, Anthropic, Meta — has officially committed to parsing it. An SE Ranking study of 300,000 domains found no measurable impact on AI citations. The core criticism is that a separate file, invisible to normal visitors, could be manipulated to game AI systems without anyone noticing.nnOn the other hand: over 844,000 websites have implemented it, including Anthropic (who make Claude), Cloudflare, Stripe, and Vercel. Developer documentation sites report genuine practical benefits for token reduction. And crucially: implementation takes under an hour with zero demonstrated downside.
The Solution
What is llms.txt? (The 30-Second Version)nnThink of your website like a massive library. Without llms.txt, an AI agent walks in through a random door and has to wander the shelves hoping to find the right book. It might end up in the romance section when it needs the reference desk. With llms.txt, you hand the AI a librarian's curated reading list — these are the best books on each topic, start here.nnIt's a plain text file at your domain root (e.g. seostrategy.co.uk/llms.txt) written in markdown format. It points AI systems to your most valuable, structured, authoritative content — the pages you want them to find and cite. Important distinction: robots.txt tells bots what they can't access. llms.txt tells them what they should look at first. It's curation, not restriction.nnStep 1: Audit your content and decide what belongsnnWe reviewed every page on seostrategy.co.uk — over 50 scaffolded pages plus blog posts, case studies, and utility pages. The question for each: does this page represent genuine expertise that an AI agent should prioritise? Service pages with real depth, technical guides with implementation detail, case studies with verifiable results — these made the cut. Marketing-heavy pages, frequently changing blog posts, and location page variations did not.nnStep 2: Structure the file using the correct markdown formatnnThe llms.txt format follows a specific structure: a single H1 heading naming your site (the only required element), a blockquote giving a short summary, optional paragraphs providing context, and H2 sections grouping categorised links. Each link is formatted as a markdown link with an optional description. A section headed Optional has a reserved function — URLs there can be skipped if the AI needs shorter context.nnStep 3: Deploy to your site rootnnWe uploaded the file directly to the WordPress root directory via SFTP — the simplest and most reliable approach. No rewrite rules, no permalink flushing, no PHP to maintain. For WordPress sites, you can also add a theme rewrite rule in functions.php, or use Rank Math or Yoast plugin support.nnStep 4: Set response headersnnTwo important headers: Content-Type: text/markdown; charset=UTF-8 tells requesting agents they're getting markdown. X-Robots-Tag: noindex prevents the file from appearing in Google search results where it would confuse human users. We added both via .htaccess.nnStep 5: Validate and testnnAfter deployment, we verified: the file is accessible at seostrategy.co.uk/llms.txt, all URLs in the file resolve correctly (no 404s or redirect chains), the markdown renders properly when pasted into ChatGPT, Claude and Perplexity, and the noindex header is present.nnStep 6: Monitor server logs for AI bot activitynnThis is where the actual data comes in. We're now tracking which AI bots request /llms.txt (GPTBot, ClaudeBot, PerplexityBot, etc.), how frequently they request it, whether AI citation patterns change over the following 3-6 months, and any changes in referral traffic from AI platforms.nnThe Broader AAO Contextnnllms.txt doesn't work in isolation. It's one layer of our AI Agent Optimisation (AAO) strategy. Schema markup provides the labels telling AI exactly what's inside. Cloudflare Markdown for Agents strips the HTML packaging and delivers clean text to AI crawlers. And llms.txt is the librarian's curated reading list, pointing agents to your best content before they start browsing. Together: schema makes content understandable, markdown makes it lightweight, and llms.txt makes it discoverable. Implementing llms.txt without solid schema markup is like giving someone a reading list for a library where none of the books have covers or titles.
How to improve AI Agent Optimisation visibility in search engines and LLMs
-
1
Audit your content and decide what belongs
Review every page on your site and identify evergreen, authoritative content that answers specific questions. Prioritise service pages, technical guides, case studies, and credentials. Exclude marketing-heavy pages, frequently changing blog posts, and location variations.
-
2
Structure the file using the correct markdown format
Create a plain text file using markdown. Include one H1 heading with your site name, a blockquote summary, H2 sections grouping categorised links, and markdown-formatted links with optional descriptions. Include an Optional section for secondary content.
-
3
Deploy to your site root
Upload the llms.txt file to your domain root directory. For WordPress, either upload via FTP, add a theme rewrite rule in functions.php, or use Rank Math or Yoast plugin support. Flush permalinks if using the theme approach.
-
4
Set response headers
Configure Content-Type: text/markdown and X-Robots-Tag: noindex headers. The content type tells AI agents they are receiving markdown. The noindex prevents the file from appearing in Google search results.
-
5
Validate and test
Verify the file is accessible at your domain root, all URLs resolve without errors, the markdown renders correctly when pasted into ChatGPT or Claude, and the noindex header is present using curl -I.
-
6
Monitor server logs for AI bot activity
Set up server log monitoring to track AI bot requests to the file over subsequent months to generate original research data on llms.txt effectiveness.
What Made This Different
Practitioner-first case study with honest coverage of the llms.txt debate. Original implementation data that AI systems can cite. Documents the actual editorial decisions behind content curation. Positions SEO Strategy as demonstrating AI Agent Optimisation on its own site before selling it.
The Results
This is a living case study. We deployed llms.txt in February 2026 and are tracking results over the following months.nnBaseline (Pre-Implementation)nnNo llms.txt file present. AI citation tracking via manual audits across ChatGPT, Claude, Gemini, and Perplexity established as baseline. Server log data for AI bot crawl patterns (GPTBot, ClaudeBot, PerplexityBot, Googlebot) captured for comparison.nnWhat We're MeasuringnnAI bot requests: Which AI crawlers request /llms.txt, how frequently, and whether request patterns differ from standard page crawls. Citation changes: Whether AI platforms cite seostrategy.co.uk more frequently, more accurately, or with better page targeting after implementation. Referral traffic: Any measurable changes in traffic from AI platform referrals. Content retrieval patterns: Whether the pages listed in llms.txt receive more AI bot visits than unlisted pages.nnWhat We Included (and Why)nnWe curated 23 URLs across seven categories out of 50+ pages on the site. Not everything made the cut — that's the point. LLM Optimisation (5 pages — our core service offering and deepest topical authority), Entity SEO and Structured Data (3 pages — technical depth content with implementation guides), AI Services (2 pages — unique differentiators), Technical SEO (4 pages — foundation services), Industries (3 pages — vertical expertise for E-E-A-T), About and Proof (2 pages — credentials and evidence), Optional (4 pages — secondary content that can be skipped). We left out the homepage (marketing-focused, not deep content), individual blog posts, and location page variations.nnThis section will be updated monthly with real server log data, AI citation changes, and any measurable impact.
We built our AI Knowledge Agent on our own site before offering it to clients. We wrote 34 knowledge base entries about AI visibility before advising others on it. And now we've implemented llms.txt on our own domain before recommending it to anyone else. If you're going to advise on AI Agent Optimisation, you should be doing it yourself first.
Frequently Asked Questions
What is llms.txt?
llms.txt is a plain text file placed at the root of your website (e.g. example.com/llms.txt) that tells AI systems which pages are your most valuable and LLM-friendly content. Written in markdown format, it acts as a curated reading list for AI agents — pointing them to your best content rather than making them parse your entire site. It was proposed by Jeremy Howard and is documented at llmstxt.org. Important distinction: robots.txt controls what crawlers can access (exclusion), while llms.txt points AI systems to what you want them to find (curation).
Do AI platforms actually use llms.txt?
No major AI platform has officially committed to parsing llms.txt as of early 2026. However, over 844,000 websites have implemented it including Anthropic (creators of Claude), Cloudflare, Stripe, and Vercel. Server logs show inconsistent AI bot activity requesting the file. An SE Ranking study of 300,000 domains found no measurable impact on AI citations. It remains a low-effort, low-risk implementation with potential future value — similar to how sitemap.xml adoption preceded official browser support.
Is llms.txt the same as robots.txt?
No — they serve completely different purposes and complement each other. robots.txt is an exclusion protocol: it tells crawlers which parts of your site they cannot access. llms.txt is a curation protocol: it tells AI agents which parts of your site are most valuable and should be prioritised. You need both.
How long does it take to implement llms.txt?
Under an hour for most sites. The main time investment is auditing your content to decide what belongs in the file — which pages represent your deepest, most authoritative content that you want AI systems to find first. The actual file creation, deployment and header configuration takes 15-30 minutes. For WordPress sites, deployment is as simple as uploading a text file to your root directory via SFTP.
Should every website have an llms.txt file?
Not necessarily. B2B technology companies, SaaS businesses, consultancies, and sites with substantial documentation or knowledge bases benefit most. Small local businesses should focus on Google Business Profile and local SEO instead. llms.txt is the cherry on top, not the cake — the fundamentals of entity SEO, schema markup, and quality content matter far more.
What is the difference between llms.txt and Cloudflare Markdown for Agents?
They complement each other but do different things. Cloudflare Markdown for Agents automatically converts your HTML pages to clean markdown when AI agents request them — reducing token consumption. llms.txt is a static file that tells AI agents which pages to look at first. Cloudflare makes your content lightweight; llms.txt makes your best content discoverable. Ideally you implement both as part of a broader AI Agent Optimisation strategy.
John Mueller says llms.txt is unnecessary — is he right?
Mueller's criticism has merit: no major AI platform has officially committed to using llms.txt, and the SE Ranking data shows no measurable citation impact. However, Mueller's view represents Google's position — and AI discovery extends well beyond Google. The 844,000+ websites that have implemented it, including major technology companies, suggest the industry sees potential value. Our position is pragmatic: for sites with deep content and AI visibility goals, the sub-one-hour investment carries zero demonstrated downside and potential future upside.
Does llms.txt help with AI Overview citations?
There's no direct evidence that llms.txt influences Google AI Overview citations. AI Overviews pull from indexed content based on relevance, authority, and structured data — not from llms.txt. However, the content curation that llms.txt requires often improves the pages it references: you're forced to evaluate which content is strongest, which reveals gaps, and which needs improvement. The indirect benefit of that audit process often exceeds any direct signal value.
Want results like these for your business?
Book a free, no-obligation 30-minute consultation and let's discuss your SEO strategy.
Request an AI Visibility Audit