An ai.txt file is a simple text file at your site's root that states, in plain language, how AI systems may use your content. The free ai.txt generator builds that file for you in seconds: you choose which AI crawlers may visit, whether your content may be used for training, and which sources AI answers should prefer, and the tool outputs a ready-to-upload file. There is no signup, no email address, and no account to create.
This guide explains what ai.txt is, how it differs from robots.txt and llms.txt, and how to write a file that actually reflects your choices. It also states the limits plainly, because ai.txt is a newer and less supported convention than robots.txt, and a guide that hides that fact does you no favours.
What ai.txt is, and what it is not
ai.txt is a proposed convention for AI usage policies. A site places a text file at /ai.txt, and the file lists AI agents, what they may do, and under what conditions. A typical file names agents such as GPTBot or ClaudeBot, states whether training use is allowed, and may point to preferred sources or contact details for licensing enquiries.
Three things ai.txt is not. It is not a replacement for robots.txt, which remains the widely supported way to control crawling. It is not a legal licence on its own, although it can state your licensing position clearly. And it is not a guarantee: an AI system that does not read ai.txt will not be bound by it, just as a scraper that ignores robots.txt is not bound by that file either. ai.txt is a clear, machine-readable statement of intent, and its value grows as more tools adopt it.
How ai.txt differs from robots.txt and llms.txt
The three files answer different questions, and a site that wants full coverage needs to understand all three.
robots.txt answers: may a crawler fetch this path? It is the oldest and most widely respected of the three, and it controls fetching at the path level for named user agents. If you want to block GPTBot from your site, robots.txt is still the first tool to use.
llms.txt answers: what does this site contain, in summary? It is a markdown file that lists a site's key pages with short descriptions, designed to help AI tools understand a site quickly. It is a discovery aid, not a permission file, and current log studies suggest AI crawlers rarely request it.
ai.txt answers: how may AI systems use this content? It covers permissions beyond fetching, such as training use, summarisation, and citation preferences. That is a wider question than robots.txt asks, and it is why ai.txt exists as a separate file rather than as an extra robots.txt rule.
In practice, a careful site uses robots.txt for crawler control, llms.txt as a summary if it wants one, and ai.txt for the usage policy. The three files do not conflict, because they answer different questions, and together they give a complete picture of a site's AI position.
Using the free ai.txt generator
Open the free ai.txt generator. The tool asks a short set of questions, and each answer becomes a line in your file:
- Which AI agents may visit? Choose all agents, a short list, or none. Naming specific agents is better than a vague statement, because a named rule is easier for a tool to follow and for your team to audit later.
- May your content be used for training? Answer yes or no. This is the decision most site owners care about, and it should be a deliberate choice, not a default you accepted without reading.
- May AI search bots cite your pages? This is separate from training. Many sites that opt out of training still want to appear in AI search answers, because those answers send visitors. Keep the two decisions separate in the file.
- Preferred sources and contact details. Add a contact address for licensing enquiries, and any source preferences you want AI answers to respect, such as preferring your official documentation over a third-party summary.
Download the generated file, read it line by line, and only then upload it to your site root. A generator output you have not read is no better than a template you copied blindly. Check that the agent names match the ones you meant, that the training answer is the one you intended, and that no line contradicts another.
What to write in each section
Agent permissions
List agents by their public names, and state the permission for each in plain words: allowed, disallowed, or allowed with conditions. If you allow an agent for search but not for training, say both things, in separate lines. A file that says only "allowed" for an agent will be read as a general permission, and you will have no record that you meant something narrower.
Training policy
State your training policy in one clear sentence, then list any exceptions. For example, a site might disallow training use for its articles but allow it for its public documentation. Exceptions are fine, as long as they are specific. A policy that says "no training, except sometimes" without naming the exception is not a policy anyone can follow.
Search and citation preferences
If you allow AI search, state whether you want citations to your original pages, and whether you prefer certain pages to be cited over others. This section is often skipped, but it is the one that most directly affects how your site appears in AI answers. A preference for your own documentation over aggregator pages is a reasonable and common choice.
Contact and licensing
Add a contact address for licensing enquiries, and a short statement of your licensing position if you have one. This does not create a legal agreement by itself, but it gives a legitimate AI provider a clear route to ask permission, which is more useful than a silent file that only says no.
Uploading and verifying the file
Upload ai.txt to the root of your domain, so it loads at yoursite.com/ai.txt. The upload method depends on your platform: a static file upload for a static site, a file in the public folder for a framework such as Astro, or a media upload for a hosted builder, if that builder allows root files at all. Some hosted platforms do not allow arbitrary root files, and in that case ai.txt may not be practical on that platform. Check the live address after upload, and read the file as served, not as generated, because a platform can change line endings or add a wrapper that breaks the format.
Verification for ai.txt is simpler than for robots.txt, because there is no widely used tester that simulates AI agents reading the file. Your checks are: the file loads at the root address, the text matches what you generated, and the agent names and permissions are exactly as you intended. Then monitor your server logs for requests to /ai.txt over the following months. If you see requests, note which agents asked, because that tells you which tools have adopted the convention.
The limits, stated plainly
ai.txt is a young convention. It does not have the adoption of robots.txt, it is not a formal standard, and there is no published list of major AI vendors that commit to reading it. That means a site should not treat ai.txt as its only AI control. Keep your robots.txt rules in place, because those are the rules the major crawlers demonstrably follow today.
ai.txt also does not stop a bad actor. A scraper that ignores robots.txt will ignore ai.txt too. Server-level controls, rate limiting, and bot management at your host or content delivery network are the tools for that problem, not another text file.
Finally, ai.txt does not settle legal questions. Whether AI training on public content is lawful, and what a file like ai.txt means in a dispute, are questions for lawyers and courts, not for a generator tool. The file is best understood as a clear statement of your position, which helps well-behaved tools respect your wishes and gives you a clean record of what those wishes were.
When ai.txt is worth adding
Add ai.txt when you have already set your robots.txt policy and you want a clearer, more detailed statement of AI usage permissions than robots.txt can express. That is the right order: robots.txt first, because it works today; ai.txt second, because it expresses more and may work better tomorrow.
Skip ai.txt for now if you have no one to maintain it, or if your platform cannot serve a root file reliably. A stale ai.txt file that contradicts your current robots.txt policy will confuse anyone who reads both, and a file that cannot be served at the root address is not a file at all. In both cases, a clean robots.txt policy alone is the better choice until the situation changes.
For most sites, though, the cost is a few minutes and the benefit is a public, readable policy that grows more useful as tools adopt it. Generate the file with the free ai.txt generator, read it, upload it, and keep your robots.txt policy as the enforceable layer underneath.
FAQ
Is ai.txt a replacement for robots.txt?
No. Robots.txt controls crawling and is widely supported today. ai.txt states broader AI usage permissions and is a newer, less adopted convention. Use robots.txt as your main control and ai.txt as a supplementary policy.
Do major AI companies read ai.txt?
There is no published commitment from the major vendors to read ai.txt in the way they commit to robots.txt. Treat ai.txt as a forward-looking statement of intent, and monitor your logs for requests to the file.
Can ai.txt stop AI training on my content?
Not by itself. It states your position clearly, but only tools that read and respect the file will follow it. For training opt-outs that work today, use robots.txt rules for training crawlers such as GPTBot, ClaudeBot, and Google-Extended.
Where does ai.txt go on my site?
At the root of your domain, at yoursite.com/ai.txt, as a plain text file. Verify it loads at that address after upload.
Is the ai.txt generator really free?
Yes. The CrawlGuards generator requires no signup and no email address. You answer a few questions, download the file, and upload it yourself.
Should ai.txt and robots.txt ever disagree?
No. Keep the two files consistent. If robots.txt blocks a crawler, ai.txt should not invite that same crawler to use the content. Read both files together whenever you change either one.
