This is the second research-led piece in our LLM Optimisation series. The first, Ranked but Not Recommended, was about search. This one is about your own site, and the plugin we maintain for it, LLMs.txt Curator, now at version 2.2.
What does “present” mean, and why is it not enough?
An llms.txt file lists the pages you want a machine to read, each with a short description. Version 1 of Curator measured whether those descriptions existed. It could tell you that 45 of 48 pages had one, and that felt like the right thing to measure.
Then we ran version 2.0 across a real product catalogue and watched it report excellent coverage on a file that was close to useless. Dozens of product descriptions opened with the same line of shop furniture, a call to action repeated across the whole catalogue, because that was the first text each page actually published. Every description was present. Almost none was useful.
That is the whole argument in one observation. Presence is a count: is there a description, is there a Markdown version, is the page in the file. Usefulness is a judgement: does what the machine receives say what the page says? A count cannot answer the second question, and a tool that only reports counts will tell you everything is fine while it is not.
Where does the gap come from?
Three places, and we found all three on live sites rather than in theory.
- The stored post is not the page. WordPress keeps the text an author typed. A page builder or a block template adds pricing tables, service grids and calls to action at render time. A Markdown version built from the stored post leaves all of that out, so the machine gets the skeleton and the visitor gets the page.
- The page is more than its content. Convert the rendered page instead and the opposite happens: menus, footers, share buttons, comment forms and “more posts” lists leak into the Markdown. A page about retrofit assessments ends up half about the site’s navigation.
- Converters lose structure. A regular-expression converter flattens a pricing table into one run-on line, so a machine reads “ServicePriceAudit” where the page said three separate things. Numbered lists lose their numbers; nested lists go flat; code loses its indentation.
And one more that is not about structure at all. A whitespace pattern in the plugin, run in byte mode, matched a byte that sits inside letters in most non-Latin scripts. On a Persian site it turned a word into a question mark and an invalid file. The description was present. The description was wrong. Nothing in a coverage count could have seen it. Saeid Afshari of kabook.ir did, reported it, and is credited in the changelog.
The Museum Guide
The physical picture we use for this is a museum.
Your website is the museum. Visitors walk the rooms and look at the exhibits, which are your pages. The llms.txt is the printed guide at the entrance: a short list of the rooms worth seeing and a line about each. A machine arriving at your site is a visitor who reads the guide first.
Version 2.0 of Curator was about the guide. Which rooms are listed, who decided, what the line about each one says, and whether the guide at the entrance is the one you approved or an old print run somebody forgot to replace.
Version 2.2 is about the labels on the exhibits. When a machine follows the guide into a room, the Markdown version of the page is the label it reads. If the label was written from the curator’s notes rather than from the exhibit itself, it describes the painting that was meant to hang there, not the one that does. If the label has the fire-exit sign and the gift-shop opening hours printed across it, the visitor cannot tell which words are about the painting. If the label has been transcribed by someone who cannot read the alphabet it was written in, it says something, and what it says is wrong.
The job of a good museum guide is not to write new labels. It is to walk the rooms, read each label standing in front of the exhibit, and say plainly where they disagree. That is what the Markdown view and the Pages checks in 2.2 do.
What did we actually observe?
These are the observations the release was built from. They are specific to the sites they were seen on, and we do not generalise from them to anyone else’s site.
The last row is a test artefact, not a client. It is in the table because it shows the failure mode cleanly: a page can be fine for a visitor and unusable for a machine purely because of what surrounds the content.
What does Curator 2.2 do about it?
It builds the label from the exhibit, lets you read the label, and says where it disagrees.
- Snapshot, not stored post. With Markdown pages on, each page in your llms.txt is fetched from your own site as a logged-out visitor, in the background. The Markdown is built from the page’s main content area, so page-builder and template content is included and the surrounding furniture is left out.
- A converter that keeps structure. Tables stay tables, lists keep their numbers and nesting, code keeps its indentation. One converter writes every Markdown address and
llms-full.txt, so they cannot disagree with each other. - A Markdown view per page. What the machine receives, laid out for reading, with how it was made: snapshot or stored content, which part of the page was used, what was removed, what was kept.
- Checks that name the problem. Pages built from stored content because a snapshot failed, and why. Pages with very little text. Unrendered shortcodes. Leftover markup. Repeated text. Uneven tables. Menus that leaked in. Observed facts, not a score.
- Never stale. A snapshot is retired the moment its page changes, and when a post is unpublished or made private every page that linked to it is retired too, so it leaves lists of posts straight away.
- Byte-safe text. One whitespace pattern that treats non-Latin letters as letters, covered by a test of every character from U+0080 to U+FFFF.
Two things it deliberately does not do. It does not write or rewrite your text; the honest answer to a bad label is a human, not a model. And it does not score pages. A score would invite the same mistake coverage invited: a number that looks like a verdict.
What does this have to do with agents?
Once the label on each exhibit is trustworthy, you can let someone else read it. Version 2.2 opens Curator to signed-in tools and agents through WordPress’s own Abilities API, read-only: list what the llms.txt publishes, get a page’s Markdown exactly as its .md address serves it, and ask what changed since last time. With WordPress’s MCP Adapter installed, an AI client such as Claude can use the same four abilities.
The rules are strict because the alternative is not worth having. The abilities answer only from the record of what you published, never from your working curation. Anything not published gets one identical “not available” answer, so nothing hidden can be enumerated. No ability can change curation, settings or what is published. Anonymous requests are refused. In the museum, the audio guide reads the labels; it does not get a key to the storeroom.
What should you do first?
- Look before you count. Open three pages’ Markdown versions, the ones you would least like a machine to misread, and read them standing in front of the page. If you use Curator, that is the Markdown view in Your llms.txt.
- Check the furniture. If your menu, footer or related-posts list appears in the Markdown, the machine is reading the building, not the exhibit.
- Check the structure. Pricing and comparison tables are where the most meaning lives and where converters lose the most. If a table reads as one line, fix the source or the converter before anything else.
- If your site is not in a Latin script, check a page by eye. A pattern bug can pass every automated test written by someone whose alphabet it does not touch.
- Then curate the guide. The entrance guide still matters: which pages, in what order, with what line. Our llms.txt guide covers that, and Curator is free.
Where this sits in the wider work: being present is the bottom of the AI Discovery Stack, and most of the Stack is about what happens after a machine has found you. But a machine that reads the wrong label learns the wrong thing, and nothing further up the Stack corrects that. Strong brands rank, get cited, remembered, recommended and dominate; a brand whose own pages describe it wrongly to machines is handing that work to someone else.
