AI Crawlers and EU Copyright: How the TDM Opt-Out Works, What Robots.txt Must Do, and the C-806/26 Case to Watch

In the EU, AI training without a rights reservation is lawful by default. That is the single most important fact in this whole area of law, and it is the opposite of what most content owners assume. You do not start with a right to stop AI companies training on your work, which you then have to defend. You start with a statutory exception that lets them train on it, and you keep that right only if you reserve it in the machine-readable way the law requires. The opt-out is the right itself.

That framing matters because it changes what a robots.txt rule is. In the United States, robots.txt is a courtesy. A crawler that ignores it is rude, not illegal. In the EU, for the text-and-data-mining (TDM) exception, robots.txt is the mechanism the law itself names as the proper way to make a reservation. The same file, the same two lines, carry a different legal weight on either side of the Atlantic. This article explains how the two pieces of EU law fit together — Article 4 of the 2019 Copyright Directive and Article 53 of the EU AI Act — what robots.txt must do to count as a reservation, and what the pending Hungarian referral, Case C-806/26, may change. Our guide to the single robots.txt rule the AI companies agreed in October 2026 covers the practical side of the new shared signal. This article covers the law that makes that signal necessary.

Article 4: the TDM exception, and the reservation that defeats it

Article 4 of Directive (EU) 2019/790, the Copyright in the Digital Single Market (DSM) Directive, creates an exception for text and data mining. TDM is the automated analysis of text and data in digital form to generate information, including patterns, trends and correlations. Training a large language model is a form of text and data mining. So is crawling a website to collect material for a training corpus. Under Article 4, that copying is lawful without the rights holder's permission — unless the rights holder has expressly reserved the use of their works for TDM purposes.

That reservation is Article 4(3), and it is the whole ballgame for EU publishers, bloggers and businesses. The conditions are precise, and they are worth quoting in plain language:

• The reservation must be made in an appropriate manner.
• For content that is made publicly available online, an appropriate manner is a machine-readable means.
• Robots.txt is the example the Directive itself gives, and it is the standard the EU AI Office has since endorsed as the primary method for web publishers to express machine-readable preferences.

The reserve is not a back-door copyright claim. It is a statutory mechanism built into the exception. Think of it this way: the EU gave AI developers a default permission to mine, and gave rights holders a default veto, if they express it in the right form. A site that does nothing has not been wronged. A site that reserves in a machine-readable way has exercised a statutory right.

Article 4 also covers the circumvention of technological protection measures. You cannot rely on robots.txt alone for content behind a paywall or login. A crawler that bypasses a paywall to mine content is violating a separate provision, not just ignoring your robots file.

One other limit is worth naming. The TDM exception covers reproduction and extraction carried out in the scope of text and data mining. It does not give a right to make a mined work available to the public, or to build a competing service from it. The law's boundary between mining and reuse is exactly what the pending Case C-806/26 referral is about. That will be covered in the final section.

Article 53: the AI Act turns the reservation into a compliance duty

Article 53 of Regulation (EU) 2024/1689, the EU AI Act, is the provision that turns the copyright reservation from a nice-to-have into a compliance obligation with fines attached. It applies to providers of general-purpose AI (GPAI) models placed on the EU market. If you build or distribute a foundation model in the EU, Article 53 requires you to do four things, and the second is the one that matters here:

1. Maintain up-to-date technical documentation on the model.
2. Put in place a policy to comply with Union copyright law, including by identifying and complying with reservations of rights made under Article 4(3) of the DSM Directive.
3. Draw up and publish a sufficiently detailed summary of the training data.
4. For models with systemic risk, carry out additional duties around evaluation, incident reporting and cybersecurity.

The second point is legally significant because it does not tell providers to respect copyright in general terms. It tells them specifically to identify and comply with Article 4(3) reservations. A provider that ignores a machine-readable reservation made through robots.txt is not just being impolite now. It is breaching a policy duty under Article 53, and its policy is supposed to be in place before it places the model on the market.

Article 53 also requires providers to have a summary of the content used for training. That summary obligation is what makes the compliance story auditable. If a provider's training corpus is documented, and your site appears in it despite a reservation, the reservation is evidence the provider can be shown to have ignored. The EU AI Office in Brussels directly supervises this obligation for large providers, and non-compliance can lead to fines of up to 15 million euros or 3 percent of worldwide annual turnover, whichever is higher.

What Article 53 does not do is create a new right for publishers. It enforces the right that Article 4(3) already gives you. The copyright reservation is the right. The AI Act is the enforcement mechanism that makes providers take it seriously. To use the mechanism, you must first exercise the right — by making the reservation in the first place.

What robots.txt must do to count as a reservation

The legal test for a valid reservation for online content is that it must be machine-readable. Robots.txt is the primary method the EU AI Office identifies, and the IETF standard draft-kelley-aipref-training-preference-signal shows how the new shared token is being formalised. But a reservation does not have to be robots.txt, and robots.txt does not automatically count in every form. A few practical rules follow from the legal test:

• Use a machine-readable, standard syntax. A statement on your human-readable Copyright page or Terms of Use is not machine-readable, and a court would likely not treat it as an Article 4(3) reservation for content published online. Put the reservation in robots.txt or another standard machine-readable form, such as the aipref vocabulary or C2PA metadata for images.
• Use the correct token. The detailed work on aipref.org from Open Future and Mozilla introduced a tdm vocabulary that lets site owners set preferences for AI use of their content. The new shared token, __usage_ai-training, designed by Anthropic engineer Kevin Kelley as draft-kelley-aipref-training-preference-signal-00 in the IETF, used as a user-agent in robots.txt, is the strongest practical form available as of late 2026. It targets AI training specifically and is supported by the four biggest AI companies.
• Cover every relevant bot. Until support for the shared token is universal, a reservation that names only GPTBot leaves ClaudeBot free to mine. Use a per-bot list too, including GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot, Meta-ExternalAgent, Applebot-Extended, Bytespider and the newer search agents such as OAI-SearchBot, Claude-SearchBot and Claude-User, or use the shared token alongside the individual blocks. A reservation that is narrow is a reservation that a provider can claim not to have seen.
• Decide training and search separately. A reservation under Article 4(3) is a reservation of TDM use, which is training. It is not by itself a reservation of other uses, such as search indexing or being cited in AI search answers. If you want to stop your content being used in generative search results, you need to block the search and retrieval crawlers separately. If you block Googlebot by mistake while trying to reserve TDM, you lose search visibility for no legal benefit. Keep your training and search decisions in separate rules, and test both.
• Use a standard, crawlable robots file. A robots.txt that is hidden behind a redirect, served with the wrong content type, or only reachable through a path the crawler does not check may not count. Keep it at your site root, serve the canonical URL, and test it with a tester that confirms the syntax is valid.

The rules are simple, but they are rules. A right that is not expressed in the form the law names is not a right a provider has a duty to respect. Our free robots.txt generator builds a file with the per-bot blocks and the shared __usage_ai-training token, and our robots.txt tester will confirm the syntax before you publish it.

What about llms.txt, ai.txt and other signals?

Outside robots.txt, three alternative signals have become important in the tools discussion, and they relate to TDM in different ways:

• llms.txt: A proposal, promoted by Mintlify's Jeremy Howard in 2024, for a Markdown file that a site places at /llms.txt to guide large language models. It is a way to tell AI systems what your site is about and how to use it. The format is not yet standardised, and many sites name both /llms.txt and /llms-full.txt which are not interchangeable. A provider that does not expect to find this file may summarise it differently, and the format is still being standardised in an IETF draft, draft-llms-txt. Use it for guidance, not as a replacement for a robots.txt reservation.
• ai.txt: A separate proposal from Spawning AI, led by Mat Dryhurst, imagined as a do-not-train list at the domain root. There is no single IETF standard for ai.txt at present. Use it if you like, but do not treat it as a substitute for robots.txt.
• C2PA metadata: The Coalition for Content Provenance and Authenticity has a machine-readable standard for tagging content provenance, and is a promising candidate for expressing TDM reservations for images that travel away from their source page. But C2PA requires a crawler to read the metadata format and honour it, which adds a layer of complexity that many AI crawlers have not yet built in. The rules for aipref are designed for crawler-readable signals, so they are the right place to define how C2PA tags should be interpreted. Where they are supported, however, they do not replace robots.txt for web content.

The key point is that the law names machine-readable means as the appropriate manner for online content. Robots.txt was the example the Directive itself gave, and it remains the reference the EU AI Office endorses. Other signals can support a reservation, but they are not the reference signal, and a provider can say it did not have a duty to look for them.

Case C-806/26: the Hungarian referral that may decide everything

On 23 September 2026, a Hungarian press consortium filed a referral to the Court of Justice of the European Union (CJEU), registered as Case C-806/26. The questions before the Court go directly to the scope of the TDM exception and the interaction with Article 4(3), and they matter to every EU publisher with a robots.txt file. The referral asks:

• Whether the Article 4 TDM exception can cover commercial LLM training when the training corpus is built from republished press content.
• Whether prompt-driven retrieval at answer time counts as text and data mining, or as a separate act that needs its own justification.
• Whether a trained model's weight representation reproduces the original works in a way that makes it an infringing copy in itself.
• Whether AI-generated summaries and other outputs are a reproduction of the mined works.
• What effect, if any, a rights reservation made under Article 4(3) has on those uses.

The timeline is long, and the ruling is likely years away. But the questions are precisely the questions an AI company would raise to argue that its training is lawful even when the material was scraped unlawfully. That is why the case matters far beyond Hungary. A ruling that the TDM exception covers commercial LLM training broadly would make the reservation mechanism even more important — it would be the only way out. A ruling that narrows the exception would remove the need for a reservation in some circumstances, but it would not remove the benefit of having one. Either way, the publishers who have made clear, machine-readable reservations in advance will be in the strongest position when the Court rules.

The case is also notable for what it says about lawful access. Article 4 requires that TDM be carried out on works to which the miner has lawful access. A crawler that ignores your robots file is bypassing the terms on which you made the work available. That is a different question from whether it has reserved its rights, and the referral asks how those two conditions interact. Writers who want the full legal analysis should read our coverage of the single rule and keep an eye on the CJEU docket for C-806/26.

What to do this week

If you publish in the EU, or your audience is in the EU, there are four steps worth taking now:

1. Make your Article 4(3) reservation, properly

Add a machine-readable reservation in robots.txt, naming the training agents individually and adding the shared __usage_ai-training token alongside them. Do it at the site root, with valid syntax. If you have not made a reservation at all, you have not opted out — you have left the TDM exception operating against you by default.

2. Test it and keep a record

Test your robots.txt file with a validator, and keep a dated copy of the file in your records. If you ever need to show that a provider ignored your reservation, the file itself, a test result and a server log showing the crawler's user-agent are the evidence. A reservation without a record is much harder to enforce.

3. Keep training and search separate

Write down two separate decisions: do you want your content used for training, and do you want it cited in AI answers. The first is the Article 4(3) reservation. The second is a separate crawler decision about OAI-SearchBot, Claude-SearchBot, Claude-User, Googlebot and Bingbot. Make both in robots.txt, label them clearly in your records, and test them independently.

4. Watch C-806/26 and the GPAI Code of Practice

The General-Purpose AI Code of Practice, which the EU published in July 2025, gives practical guidance on Article 53 compliance and endorses robots.txt as the primary mechanism for expressing machine-readable preferences. Providers who follow the Code have a stronger claim to compliance, and providers who ignore robots.txt have a weaker one. That Code and the pending C-806/26 ruling are the two documents that will shape what your reservation means in practice over the next two years. Check both every few months.

Frequently asked questions

Is AI training on my EU website lawful without my permission?

Yes, by default, under Article 4 of the DSM Directive. Text and data mining is a statutory exception to copyright. It becomes unlawful only if you have expressly reserved your rights under Article 4(3) in a machine-readable manner, such as through robots.txt, and the miner ignores that reservation.

Does robots.txt really have legal force in the EU?

In the sense that matters, yes. For online content, Article 4(3) says a reservation must be made in an appropriate manner, and for material made publicly available online that manner is a machine-readable means. Robots.txt is the example the Directive gives and the primary method the EU AI Office endorses. A properly made reservation engages the Article 53 duty on GPAI providers to identify and comply with it.

What is the difference between Article 4 and Article 53?

Article 4 of the 2019 Copyright Directive creates the TDM exception and the reservation that defeats it. Article 53 of the EU AI Act requires GPAI model providers to have a copyright compliance policy that identifies and complies with Article 4(3) reservations. Article 4 is the right. Article 53 is the enforcement.

Will a robots.txt reservation stop my content appearing in ChatGPT answers?

Not on its own. A TDM reservation covers training. Citations in AI answers are produced by search and retrieval crawlers, such as OAI-SearchBot and Claude-SearchBot, which are controlled separately. If you want to disappear from AI answers as well, block those crawlers in your robots.txt as a separate decision.

What is Case C-806/26 and why does it matter?

Case C-806/26 is a Hungarian press consortium's referral to the CJEU, filed on 23 September 2026. It asks whether the TDM exception covers commercial LLM training, prompt-driven retrieval and AI-generated summaries, and what effect an Article 4(3) reservation has on those uses. The ruling is likely years away, but it may determine the scope of the TDM exception for the whole EU.

Should I rely on llms.txt or ai.txt instead of robots.txt?

No. Use them as supporting signals if you like, but robots.txt is the reference the law names and the EU AI Office endorses. A reservation that exists only in llms.txt or ai.txt is a reservation a provider can argue it was not required to look for. Put the reservation in robots.txt first.