There are two completely different moments when your content meets a large language model, and almost every confused argument about AI visibility comes from collapsing them. One is runtime, what the AI fetches live when it is answering a question, where clean structure and a curated llms.txt help. The other happens long before any prompt: the model forming its standing knowledge of your brand from the data it was trained on. This guide is about the second moment, because it is the one that decides whether the model knows you at all, and it is governed by filters most SEO advice never mentions.
Models do not learn the web. They learn a filtered slice of it.
When a model is trained, it is not fed the internet as-is. Frontier labs start from enormous web crawls, Common Crawl indexes on the order of 250 billion pages, and the raw material is, in the words of people who build these systems, genuinely dirty: broken sentences, boilerplate, duplicate paragraphs repeated thousands of times, and a great deal no one would want a model to imitate. So before training, that crawl is aggressively filtered, deduplicated and quality-scored. Only a fraction survives. Whether your brand makes it into the model’s memory is decided here, at the filter, not on your own website.
The inclusion filter trusts what trusted sources reference
Here is the part that should change how you think about authority. One of the standard techniques for quality-filtering training data is disarmingly simple. Labs take the set of pages referenced by a trusted source, classically, the outbound links from Wikipedia, and train a classifier to recognise content that resembles those references versus content from the random web. Then they keep more of what looks like the trusted set and discard more of what does not. Content the trusted web already points to is upweighted into the training data. Content nobody references is more likely to be filtered out before the model ever sees it.
Read that back as a strategy and it is your whole corroboration thesis, stated as a mechanism rather than a belief. Being referenced, cited and discussed by sources the filter already trusts is not a soft reputational nice-to-have. It is, quite literally, the gate that determines whether your brand survives into the model’s parametric memory. This is the engine underneath entity corroboration: you do not earn training-corpus representation by adding files to your own site, no file you control touches this layer, you earn it by being part of what the trusted web corroborates.
The exclusion filter reads templated content by its signature
Getting referenced gets you considered. Not getting filtered out as junk is the other half, and the mechanism there is more sophisticated than simple spam rules. Google’s own researchers have published work describing how automated, templated text leaves a distinct mathematical signature in embedding space, detectable with models like Sentence-BERT, and how detection has moved from scoring pages one by one to spotting clusters of content that sit too close together because they were mass-produced from the same template. Sentence-BERT has existed for around seven years and reduces the work of finding the most similar passages from roughly 65 hours to about 5 seconds, the kind of efficiency that makes signature detection viable at web scale.
Read against our Footprint vs Fingerprint framework, the implication is precise. A footprint, content a competitor could regenerate tomorrow from the same prompt, is a crowded region of embedding space that templated, near-identical pages pile into. A fingerprint is a position only your entity could occupy, backed by record nobody else has. The same embedding geometry runs three jobs at once: deciding what gets retrieved and cited, accumulating retrieval gravity for distinctive entities, and flagging templated sameness. Distinctiveness in that space is the single property doing all three. The Google research targets coordinated video spam specifically, and the application to web text is raised rather than confirmed, but it establishes the capability and the direction of travel, and the direction is away from rewarding content that looks like everything else.
Why under-referenced brands do not just get omitted, they get invented
The stakes are sharper than absence. There is a well-supported hypothesis among researchers about where hallucination comes from, and it bears directly on weakly-corroborated brands. When a model is fine-tuned to be a helpful assistant, it is taught to produce answers in the right shape. If you then ask it about an entity it never properly learned during training, it does not reliably say it does not know. It generates something plausible-sounding in the correct format, a confident answer assembled from pattern, not knowledge. So a brand that never accumulated enough third-party corroboration to survive the training filter is not merely left out of AI answers. It is the brand the model is most likely to make things up about: wrong founding dates, invented capabilities, confident misattributions, because there was never a solid representation to anchor against.
That reframes the cost of weak corroboration. It is not just lost visibility. It is the model speaking about you, incorrectly, with confidence, to your buyers.
What this means for the work
Two practical consequences, both of which separate this layer from the runtime layer.
First, the lever here is off-page and durable. You influence the training filter by becoming something the trusted web references, consistent entity data across the sources that feed knowledge graphs, genuine third-party citations, presence in the reference-grade places (Wikipedia, Wikidata, Crunchbase) the filters lean on. None of it is a setting you toggle on your own site. It compounds slowly and survives algorithm changes, because it is anchored in what others say about you, not what you say about yourself. This is the same property our Retrieval Gravity and Editorial Selection work is built on, viewed from the training side.
Second, on-site AI files are not the answer to this problem. A curated llms.txt, clean schema and a fast, agent-readable page all help at runtime, when an agent fetches you live, and they are exactly what the Agentic Browsing audit measures. They do nothing at the training filter, which never consults your preferences. Knowing which lever moves which layer is the difference between effort that compounds and effort that is merely tidy.
A falsification note, because this is a mechanism account, not a law: this describes the dominant public technique for quality-filtering pre-training data; specific labs use proprietary variations and some are moving toward other signals. If frontier labs publicly shift away from reference-based quality classifiers as the primary filter, the mechanism here would need revising, though the strategic conclusion, that corroboration drives training-corpus representation, is supported independently by how knowledge graphs and canonical entity stores are built. We will revise this guide if the public methods move. For the runtime counterpart, how a system characterises your entity when it answers, see Teaching AI Who You Are.
