What our own site exposes to AI, and what it turns away

The site biencite.fr serves ten public machine-readable documents, lets the answer engines' crawlers in, turns away two crawlers whose job is to build a training corpus, and states its usage preferences in an HTTP header sent with every page. None of this asks for your trust: every address named in this article opens in a browser, with no account and no tooling.

Why publish an inventory of your own site?

A service that grades other websites on AI visibility should accept being graded itself. Biencité sells a diagnostic built on 19 signals across three axes: Visible, Citable, and Agents. Publishing what our own domain exposes is the only way to make that promise checkable by a reader who has no account and owes us no benefit of the doubt.

The exercise has a second use, more valuable to the reader than to us: it hands you a list of addresses to compare against your own. Open the same paths on your domain and see which ones answer. An error page means the document is not there.

Which documents does biencite.fr serve to AI today?

The domain www.biencite.fr answers on ten addresses written for machines rather than browsers. Here is the full list, as it stood on September 1, 2026.

• /robots.txt: access rules, crawler by crawler, plus the address of the sitemap.
• /sitemap.xml: the sitemap, generated automatically, covering the French and English pages.
• /llms.txt: the short summary, roughly 1,800 bytes. What the product does, who it is for, and what it does not promise.
• /llms-full.txt: the full context, roughly 10 kilobytes, regenerated hourly from the product's own data.
• /auth.md: the agent access guide, in Markdown, stating which interface is open and on what terms.
• /.well-known/api-catalog: the catalog of exposed services, in linkset format.
• /.well-known/agent-skills/index.json: the skills index. One entry today, the public scan, no authentication.
• /.well-known/mcp/server-card.json: the capability card for MCP clients.
• /.well-known/oauth-protected-resource: the authentication framework for agents.
• /.well-known/security.txt: the security contact and its expiry date.

An eleventh check is not an address but a negotiation. Ask for the /en/methode page while stating that you would rather have Markdown, and the server returns a Markdown version of the page instead of HTML. That is what the Biencité reference calls the agent-readable version: the same content, stripped of layout, for a machine that only needs the text.

Which crawlers do we let in, and which do we turn away?

The robots.txt file on biencite.fr names and allows GPTBot and OAI-SearchBot from OpenAI, ClaudeBot and anthropic-ai from Anthropic, and PerplexityBot, across the whole site except the technical routes and the customer area. Two crawlers are refused on the entire domain: CCBot, the Common Crawl collector, and Google-Extended.

The Content-Signal HTTP header, sent with every page, carries the same intent in a second form that any crawler can read: search=yes, ai-input=yes, ai-train=no. In plain English: index this site, use its pages to build an answer, do not train a model on it.

Does refusing Google-Extended cost you rankings in Google?

No, and Google says so in its own documentation: Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal there. The token governs the training of Gemini models and their grounding on external sources, nothing else.

The refusal is not free, though, and that is the part this industry tends to skip. Google files training and grounding under the same token: refusing one refuses the other. Biencité accepted that price. A company whose priority is showing up in Gemini's answers will decide the opposite way, and will be right to.

Is a statement of preference the same as a lock?

No, and the format's own author says so. Cloudflare, which published the Content Signals Policy on September 24, 2025, states plainly that these signals express preferences, that they are not technical countermeasures against scraping, and that some companies may simply ignore them.

That is exactly the subject of our article "Your robots.txt is not a lock: AI bots walk past it, and OpenAI says so". A robots.txt file and a Content-Signal header say what you accept; they impose it on nobody. Publishing them is still worth doing, for a mechanical reason: the companies that do honour preferences can only honour the ones you have written down.

Why an llms.txt file rather than an about page?

Because an about page is built for a human reader and an llms.txt file is built for a machine with a limited reading window. The proposal, published by Jeremy Howard on September 3, 2024, starts from a plain observation: web pages are built for people, with navigation, ads and JavaScript, which makes extracting the substance hard.

The strongest argument for the format is not theoretical. The AI labs publish one for their own developer documentation: OpenAI, Anthropic and Gemini each serve an llms.txt, as the proposal's own site records. The one on biencite.fr fits on a page and says three things: what the product does, who it is for, and what it does not promise.

What is in the layer written for agents?

Four files, one of which exists mainly to say what does not exist. The service catalog announces a single public interface, the scan, with no authentication. The skills index describes that same interface in a format an agent reads without guessing.

The last of the four, the authentication framework for agents, declares an empty list of authorization servers and an explicit note: the public scan needs no account, account features require a human sign-up, and there is no OAuth for agents. A document that exists to say "there is nothing here" beats a missing one: the agent gets a clear answer instead of an error to interpret.

What does this inventory not prove?

It proves no citation. A perfectly equipped site is not a cited site: it is an eligible one. That is the limit we keep repeating in "What an AI visibility scan actually measures, and what it cannot", and it applies to our own domain exactly as it applies to yours.

The inventory does not prove these files are read, either. Biencité measures what a site exposes; nobody measures what an engine then does with it. The published method states what each signal checks, and a report always says how many signals could actually be measured, because an unreachable document leaves the calculation instead of counting as a zero.

When would copying this setup be a waste of time?

More often than this industry likes to admit. A layer of machine-readable files organises content that already exists; it creates none. Four situations where the effort belongs elsewhere.

• Your site has nothing to cite: three commercial pages and no substance. Write first, equip second.
• Nobody is looking for you: with no reputation and no mentions elsewhere on the web, a flawless llms.txt will trigger no answer.
• Your platform locks the root of the domain: some of these documents will stay out of reach, and should then be marked impossible rather than counted against you.
• Your budget has better work to do: if your product pages are thin or your business listing is neglected, that is where the money goes first.