On 30 October 2026, Google, OpenAI, Microsoft and Anthropic agreed to support a single shared robots.txt signal that lets any website owner opt out of AI training across all four companies at once. Instead of writing a separate block for every crawler, you will soon be able to add one entry to your robots.txt file and have all four treat it as a do-not-train instruction. It is the biggest change to crawler control since robots.txt itself, and it matters for every site owner who has been maintaining a growing list of per-company blocks.
Here is the short version: the signal is a policy token called __usage_ai-training. You add it as a user-agent in robots.txt and disallow everything under it. Once each company finishes rolling out support over the next several months, their training crawlers will read that one entry and stay away. But there are important limits. The standard is still a draft, not a finished rule. It covers AI training only, not AI search answers. And the four companies are moving at different speeds, with Google following next year and Meta expressing support without committing to implement it.
What exactly was announced on 30 October 2026
The four companies said they will back a shared web crawling standard that lets site operators block AI developers from using their content to train generative models with one robots.txt entry. The announcement was made on 30 October 2026, and was covered in detail the same week, with further reporting on the token itself. The token is defined in an Internet-Draft submitted to the Internet Engineering Task Force (IETF), the non-profit that maintains the Robots Exclusion Protocol, the standard behind the robots.txt format. The draft, draft-kelley-aipref-training-preference-signal-00, was submitted by Kevin Kelley, an engineer at Anthropic, on the same day as the announcement. The token draws on the aipref vocabulary draft, A Vocabulary for Expressing AI Usage Preferences, co-authored by Paul Keller of Open Future and Martin Thomson of Mozilla.
Today, a site owner who wants to opt out must edit robots.txt with a separate entry for each AI provider, and for each scraper when a provider runs several. Miss one, and that company's crawler keeps coming. The new standard replaces that long list with one line. The four companies have sent the draft to the IETF, which means it is now going through the internet's normal standards process rather than being a private deal between four firms.
How the single rule works
The entry looks like this:
User-agent: __usage_ai-training
Disallow: /That is the whole thing. You do not need a new file, a vendor dashboard, or a legal agreement. You add those two lines to the robots.txt file at your site root, and once support is live at each company, all four will treat it as a training opt-out. Path-matching rules from RFC 9309 still apply, so you can add Allow: lines for specific directories you want to remain accessible to training crawlers, while blocking the rest.
Think of it the way one early report put it: before this rule, you had to tell every library assistant separately that your work was off-limits. Forget one, and they kept scanning your site. The shared signal is like putting one universal Do Not Enter sign on your front door.
If you already block GPTBot, ClaudeBot, Google-Extended and the rest individually, keep those rules in place for now. The new token does nothing until each company ships support for it. Your existing per-bot entries are the only thing actually working today.
Who is doing what, and when
The timelines differ by company, and the differences matter for your planning:
- Microsoft and OpenAI plan to add support in the coming months.
- Google will follow next year.
- Anthropic submitted the draft itself, and is behind the standard.
- Meta has expressed support for the idea without committing to implement it.
As of this writing, the draft carries no mandatory authority. It is a proposal working its way through the IETF. That is normal for internet standards, but it means you should not treat the token as a working block today. Add it now if you like, so it is in place when support arrives, but do not remove your existing per-bot rules until each company confirms it is reading the new token.
Microsoft also said it will offer a crawling setting called AI Grounding for cases where an AI model visits a page to answer a user's question rather than to collect training data. That setting would let operators block those requests, including from third-party tools built on Microsoft technologies. In other words, the companies are starting to separate two different uses of your content: training a model, and grounding an answer. The new token covers the first. Grounding is a separate conversation.
What the new rule does not cover
This is the part most early coverage glosses over, and it is the part that will cause the most confusion. The single rule covers AI training. It does not cover the use of your pages in generative search answers.
If you want your content to stop appearing in ChatGPT answers, Google AI Overviews or Copilot responses, the training opt-out will not do that. Those answers are produced by search and retrieval crawlers, not training crawlers. OpenAI's OAI-SearchBot, Anthropic's Claude-User and Claude-SearchBot, Google's regular Googlebot, and Microsoft's Bingbot are separate user-agents with separate jobs. Blocking training while allowing search crawlers is a deliberate design choice: many publishers want to be cited in AI answers while refusing to let their work train the underlying models. The new standard preserves that split.
The practical consequence is simple. After the token is live, your robots.txt will express two different decisions: a training decision (the new shared token) and a search decision (the individual search and answer-engine bots). Do not confuse the two, and do not assume one covers the other.
Why the big four agreed now
The timing is not an accident. The European Union's AI Act, in Article 53, requires providers of general-purpose AI models placed on the EU market to maintain a copyright compliance policy and to identify and comply with reservations of rights made under Article 4(3) of the EU's 2019 Copyright Directive. Google explicitly cited alignment with the EU's General-Purpose AI Code of Practice, which identifies robots.txt as the primary method for web publishers to express machine-readable preferences under the AI Act. The EU AI Office in Brussels directly supervises this obligation for large providers, and fines can reach 15 million euros or 3 percent of worldwide annual turnover.
There is also a standards logic. Each company maintaining its own opt-out token creates work for publishers and confusion for everyone. A shared token reduces the cost of opting out, which makes opt-outs more common, which in turn makes the whole system more defensible. The companies get a cleaner compliance story; publishers get a simpler file to maintain. The losers are nobody, which is why the announcement landed with so little friction.
And the legal picture is still moving. On 23 September 2026, a Hungarian press consortium filed a referral to the Court of Justice of the European Union, Case C-806/26, asking whether the EU text-and-data-mining exception covers commercial LLM training, prompt-driven retrieval and AI-generated summaries. The ruling is likely years away, but it may determine whether voluntary signals like this token satisfy the Article 4 legal threshold, or whether AI companies need full licences regardless. The token is the practical answer for today; the court will give the legal answer later.
What to do this week
You do not need to wait for the standard to finish. There are four concrete steps worth doing now:
1. Audit what you block today
Open your robots.txt file and list which AI crawlers you currently block. Most sites that set this up a year ago have a stale list: GPTBot and ClaudeBot blocked, but newer crawlers like Meta-ExternalAgent, Applebot-Extended or Bytespider missing. If you cannot remember what is in the file, that is the first problem to fix.
2. Add the new token alongside your existing rules
Adding the __usage_ai-training entry now costs nothing and means the opt-out is already in place when each company ships support. Keep every existing per-bot block until you have confirmation that the company reads the new token. Removing working blocks for a draft standard would be a mistake.
3. Decide your training policy and your search policy separately
Write down two answers. First: do you want your content used to train AI models? Second: do you want your content cited in AI answers? Many publishers answer no to the first and yes to the second. Your robots.txt should reflect both answers, with the training token for the first and individual search-bot rules for the second. If you block OAI-SearchBot by accident while trying to block training, you lose ChatGPT citations for no reason.
4. Watch the rollout dates
Microsoft and OpenAI are expected in the coming months; Google next year. When each company confirms support, test it. Fetch your robots.txt as that company's crawler would see it, and confirm the token is being read. Standards roll out unevenly, and a token that works for one company may not yet work for another.
How this fits with the tools you already have
If you manage robots.txt by hand, the new token is two lines in a file you already edit. If you use a generator, the token will need to be added as an option once it is final. Our free robots.txt generator already builds per-bot blocks for GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and the rest, and you can test any file before it goes live with our robots.txt tester. Use the tester after adding the new token to confirm your file is still valid: a typo in robots.txt can silently break every rule in the file, including the ones that were working.
For the wider picture, our guide to blocking AI crawlers with robots.txt explains the per-bot rules in detail, and our complete list of AI crawlers keeps track of which bots exist and what each one does. The new token does not replace that knowledge; it sits on top of it.
The bottom line
A single robots.txt entry that opts your site out of AI training at four companies is a genuine improvement. It replaces a fragile, per-company list with one shared signal, and it has the weight of the EU AI Act behind it. But it is a draft, not a switch you can flip today. It covers training, not search answers. And it arrives on four different schedules. The right move is to add the token now, keep your existing blocks, separate your training decision from your search decision, and check back as each company ships support. The publishers who do that will have the simplest robots.txt files on the web, and the strongest position when the standard goes final.
Frequently asked questions
Is the __usage_ai-training token working right now?
Not yet. It was announced on 30 October 2026 as an IETF Internet-Draft. Microsoft and OpenAI plan to add support in the coming months, Google will follow next year, and Meta has not committed to implementing it. Add it now so it is in place, but keep your existing per-bot blocks until each company confirms support.
Does the new rule stop my content appearing in ChatGPT or AI Overviews?
No. The token covers AI training only. Citations in AI answers come from search and retrieval crawlers such as OAI-SearchBot, Claude-SearchBot and Googlebot, which are controlled separately. Blocking training does not remove you from AI answers, and that is by design.
Do I still need to block GPTBot, ClaudeBot and Google-Extended individually?
Yes, for now. Those are the rules that actually work today. The shared token is a replacement for them in the future, not a substitute while support is still rolling out. Remove individual blocks only after the relevant company confirms it reads the new token.
What happens if I add the token and a company never supports it?
Nothing bad. An unrecognised user-agent entry in robots.txt is simply ignored by crawlers that do not know it. The two lines are harmless for any bot that does not support the token, and useful for every bot that does.
Does the token have legal force in the EU?
It is a strong fit for EU law rather than a law itself. Under Article 4(3) of the EU Copyright Directive, a rights reservation for online content must be expressed in a machine-readable manner, and the EU AI Act's Article 53 obliges general-purpose AI providers to respect such reservations. The token is exactly that kind of machine-readable signal. Whether courts ultimately treat it as sufficient is still an open question, with cases pending before the Court of Justice of the European Union.
Should small sites bother with this?
Yes, and it is easier for small sites than for large ones. Two lines in robots.txt is the cheapest opt-out ever offered. Large publishers have legal teams and complex crawler policies; a small blog or shop can now express the same training decision with the same two lines. If you care about your content not training models, add the token and keep your existing blocks.
