Your robots.txt is not a lock: AI bots walk past it, and OpenAI says so

A robots.txt file is a request, not a lock, and the two largest players in generative search now state this in their own documentation. OpenAI documents that robots.txt rules "may not apply" when a visit is triggered by a ChatGPT user. Google documents that its user-triggered fetchers "generally ignore" robots.txt rules. A business that believes it has closed the door to AI has not closed it, and a business that believes it has opened that door may not have opened it either.

What does OpenAI's documentation actually say?

OpenAI's public documentation names three bots with three different rule sets, and places only two of them under robots.txt control. GPTBot collects content to train OpenAI models. OAI-SearchBot crawls sites so they can surface in ChatGPT's search features. Both of these can be controlled from robots.txt. The third bot, ChatGPT-User, fires when a person asks a question in ChatGPT or triggers an action toward a website. For that one, OpenAI writes that robots.txt rules "may not apply", on the grounds that the action was initiated by a human rather than by automated crawling.

Google publishes the same distinction in its Search Central documentation, on a page dedicated to user-triggered fetchers. Google states there that these fetchers "generally ignore" robots.txt rules, precisely because a person requested the page. Google-Agent appears on that list. Anthropic holds the opposite position and documents it as well: its support page states that all three of its bots, ClaudeBot, Claude-User and Claude-SearchBot, honor standard robots.txt directives.

How many bots actually ignore the file?

Roughly 15% of identified AI fetcher bots on European sites reached URLs those sites had marked as disallowed, according to TollBit's State of the Bots report covering the first half of 2026, as reported by Search Engine Journal on August 14, 2026. Three bots account for most of it: ChatGPT-User, Bytespider and Youbot each reached disallowed pages on close to half of the European sites that explicitly named them, with ChatGPT-User reaching the largest number of sites.

One methodological caveat matters before you apply this to your own business. TollBit's panel is made up of content publishers, not retailers or service firms. Those figures describe accurately what happens to news and media sites, and they suggest a direction of travel for everyone else without measuring it directly. The verifiable fact does not depend on any panel at all: it is written in OpenAI's documentation and in Google's.

Why do European sites block less than North American ones?

European sites disallow these bots far less often than North American sites, which mechanically increases their exposure. According to the same TollBit report for the first half of 2026, Claude-User is blocked by 9% of European sites against 26% of North American sites, and Perplexity-User by 13% against 26%. The gap does not mean European sites are careless. It mostly means the question has not yet been decided inside most European companies, because it has not yet been asked.

What does this change for a small business?

For the overwhelming majority of small and mid-sized businesses, this news changes nothing about protecting content, and a great deal about the opposite risk: blocking without knowing it. A ten-page brochure site, a restaurant menu or a list of services is not an archive worth walling off. A small business rarely suffers from being read too much by AI systems. It suffers from not being read at all. And the rules that deny access usually arrived through a default setting, a security plugin or a checkbox ticked by an agency, never through a deliberate decision by the owner.

One concrete deadline makes that risk immediate. Cloudflare announced on July 1, 2026 that from September 15, 2026, bots in the training and agent categories will be blocked by default on pages that display advertising, while search bots stay allowed by default. The new default applies to new domains onboarding to Cloudflare. Cloudflare also notes that multi-purpose crawlers, the ones combining search with training, are allowed or blocked according to all of their behaviors, and are therefore blocked for a customer who chose to block training.

The cost of getting this wrong is measurable on the search side already. Ahrefs, studying 300,000 keywords and comparing December 2023 with December 2025, found that when an AI Overview is displayed, the click-through rate of the number one organic result is 58% lower than on comparable searches without one. That study was published on February 4, 2026. A site that answers a question well but cannot be retrieved by the systems writing those answers loses twice: once in the summary, once in the click.

Which bots should you tell apart before deciding anything?

Three families of bots exist, with three different commercial consequences, and confusing them is the single most common source of bad decisions.

• Training bots, such as OpenAI's GPTBot or Anthropic's ClaudeBot, take your content to train a model. Blocking them removes you from no AI answer at all: this is the one block that costs you no visibility.
• Answer-index bots, such as OpenAI's OAI-SearchBot, feed the search features of AI assistants. Blocking them removes you from the pool of sources those assistants can draw on.
• User-triggered fetchers, such as OpenAI's ChatGPT-User or Google's Google-Agent, retrieve a page because a person asked a question. These are the ones OpenAI's and Google's own documentation describes as able to bypass robots.txt.

How do you find out what is really happening on your site?

Only your server logs or your CDN logs show what actually reached your site; robots.txt only shows what you asked for. Search Engine Journal made exactly that distinction in its August 14, 2026 article, and it is the only honest way to answer the question "are AI systems reading my site". An audit of public files, robots.txt included, measures something different and measures it well: what your site declares to answer engines, which is its eligibility to be read and cited. Both measurements are useful. They simply answer different questions, and any vendor selling you the first while implying it proves the second is selling you a confusion.

This news extends a subject already covered on this blog. Our article "How much does an AI visibility audit cost in 2026?" sets out what an audit of this kind can and cannot establish, and what a realistic budget looks like. Our French-language piece on llms.txt covers the other half of the equation: what you can declare voluntarily to answer engines, where robots.txt can only forbid. The shared lesson fits in one sentence: what you publicly declare should be true and deliberate, because it is the only part of the equation still under your control.