The short answer
If you run a Django site and you want AI crawlers to stay out of your training data, put a robots.txt file at the root of your domain, at yourdomain.com/robots.txt, and list the AI bot tokens you want to block. There is no single Django setting that does this for you. The file itself does the work, and Django is just the delivery system that has to serve it from the right place. Get the file to the domain root, test the live URL after every deploy, and check your server logs to see whether the bots actually stopped coming.
That last step matters more than most guides admit. A robots.txt file that lives at a static files path, or one that gets overwritten by a deploy, protects nothing. This guide walks through the reliable ways to serve the file on Django, which bots to block and which to leave alone, how staging and production should differ, and the mistakes that quietly undo the whole setup.
Why robots.txt is the right first move on Django
Django is a Python web framework used to build everything from small blogs to large content platforms. It gives you a lot of control over URLs, views, and templates, but it does not ship with a built-in robots.txt manager. That is normal. Robots.txt is a web standard, not a framework feature. Every framework, Django included, ends up solving the same problem: make sure a plain text file is reachable at the root of the domain.
Robots.txt works as a request, not a lock. Well-behaved crawlers read it before they fetch your pages and follow what it says. The major AI crawlers, including the ones run by OpenAI, Anthropic, Perplexity, and ByteDance, state publicly that they respect robots.txt. That makes it the cheapest, simplest, and most widely supported first layer of control. It will not stop a scraper that ignores the rules, and it should never be your only protection for private data. For public pages that you simply do not want used for AI training, though, it is the correct starting point.
Think of it in layers. Robots.txt tells polite bots what to skip. Real access control, like login requirements on admin pages and private areas, keeps people and bots out of places they should never reach at all. Rate limiting and edge rules can add a harder layer on top. Start with robots.txt, verify it, and then decide whether you need more.
Know the bots before you block them
Blocking every bot sounds safe, but it usually costs you traffic and visibility you actually want. The smarter move is to split bots into two groups: search bots that bring you visitors, and training bots that feed AI models. You can allow the first group and block the second. That split is the whole point of a careful robots.txt file.
Training bots to block
GPTBot is the crawler OpenAI uses to gather training data. Blocking it does not remove your site from ChatGPT answers, because ChatGPT citations come from a different crawler. If your goal is to keep your content out of model training, GPTBot is the token to block.
ClaudeBot is Anthropic's crawler for training and improving its models. CCBot is the crawler from Common Crawl, whose datasets are widely used to train AI systems. Bytespider is ByteDance's crawler, used for training data collection. PerplexityBot is the crawler behind Perplexity's AI search and answer engine. If you do not want your pages feeding these systems, list each token by name in your file.
Search and citation bots to think twice about
OAI-SearchBot is separate from GPTBot. It powers search and citation features in ChatGPT, which means blocking it can remove your pages from ChatGPT answers and links. Some site owners want that. Others want the citations and the traffic that comes with them, but not the training use. Those are two different decisions, and robots.txt lets you make them separately. Block GPTBot, allow OAI-SearchBot, and you get citations without training use, at least as far as these published tokens go.
Google-Extended and Applebot-Extended are not crawlers themselves. They are tokens that Google and Apple use to signal AI-related uses of already crawled content, such as training and features built on top of search data. Blocking Google-Extended does not block Google Search. It signals that you do not want your content used for certain AI purposes. The same logic applies to Applebot-Extended. If you block these tokens, be clear about what you are asking for: you are opting out of AI uses, not out of search indexing.
The practical rule is simple. Write down what you want before you write the file. Do you want search visibility? Do you want AI citations? Do you want to stay out of training data? Your answers decide which tokens go in the block list and which stay out. A file written without those answers tends to block too much or too little.
Step-by-step: get robots.txt live on Django
Follow these steps in order. Each one fixes a specific way Django setups quietly fail at this job.
- Decide your policy first. List the bots you will block, the bots you will allow, and any URL patterns you want bots to skip, such as search result pages and filtered listings. Keep this list short and deliberate. A long, copied file full of tokens you have never heard of is harder to maintain and easier to get wrong.
- Write the file as plain text. Use the sample later in this guide as a starting point and adjust the token list to match your policy. Keep one rule per line, use the exact bot token spellings, and avoid clever formatting. Robots.txt parsers are strict about simple things, and a typo in a token name means that bot is not blocked.
- Serve it from the domain root. This is the step that trips up Django projects most often. Your file must be reachable at yourdomain.com/robots.txt, not at a path under your static files setup. In Django projects, static files are collected and served from the static setup during deployment. That setup is built for assets like stylesheets and images, and its URLs sit under a static path. A robots.txt file placed there will load fine at that static path and still fail its real job, because crawlers only look at the domain root. After you deploy, open yourdomain.com/robots.txt in a browser and confirm the file loads there. If it does not, the rest of the setup does not matter yet.
- Use a URL route when the root file is hard to place. Another reliable route is to map a URL route directly to the robots.txt path so that a request for /robots.txt returns the file's text as a plain text response. Described in plain terms, you add a route for that exact path and point it at a handler that returns your robots.txt content. This approach survives static file collection because it does not depend on where static files end up. It also makes it easy to serve different content on staging and production, which matters a lot, as the next section explains. Whichever route you choose, the test is the same: the live root URL must return the file.
- Test the file with a checker. Paste your live URL into a robots.txt checker and confirm the rules parse as you expect. A free checker such as the robots.txt tester can show you which rules apply to a given bot and URL. If you are starting from scratch, a robots.txt generator can help you build a clean file before you deploy it. Test before you declare the job done, not after a bot has already crawled pages you wanted blocked.
- Deploy, then test again on the live domain. Static file collection and deploy pipelines can overwrite or miss files. A file that worked on your local machine may not survive the pipeline to production. After every deploy that touches static files, routing, or server settings, open the live robots.txt URL and read it. This takes ten seconds and catches the most common failure mode for Django sites.
- Check your server logs. Robots.txt is a request, and requests can be ignored. Look in your server logs for the bot tokens you blocked, such as GPTBot, ClaudeBot, and CCBot. If visits from those tokens drop to near zero after your file goes live, the file is working. If they keep fetching your pages at the same rate, either the file is not at the root, the token is misspelled, or that crawler is not respecting the file, and you need a harder layer of control. Logs are the only honest proof here. Do not assume the file works because it exists.
A complete robots.txt sample for Django sites
Here is a sample that allows search engines, blocks the main AI training crawlers, and keeps bots out of filtered and search URLs. Adjust the token list to your own policy before you use it. The disallow rules for query-string URLs are there to save crawl budget: filter and search pages create endless URL combinations, and bots do not need to crawl all of them.
User-agent: *
Disallow: /search/
Disallow: /*?*filter=
Disallow: /*?*sort=
Disallow: /admin/
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
Sitemap: https://yourdomain.com/sitemap.xmlRead the sample carefully. The first block applies to all bots and keeps them out of search pages, filtered listings, and the admin path. The later blocks apply to specific AI tokens and block those bots from the whole site. OAI-SearchBot is deliberately not blocked here, because this sample assumes you want ChatGPT citations while opting out of training. If you do not want citations either, add a block for OAI-SearchBot too. Replace yourdomain.com with your real domain, and remove any rule that does not match your site. A rule for a path you do not have is harmless, but a rule you copied without understanding is how sites accidentally block their own content from search.
For the official format rules, Google's robots.txt documentation is the reference worth reading: Google's robots.txt specification. If you want background on how Django handles static files during deployment, the Django docs explain the setup here: Django static files guide.
Staging versus production: do not use the same file
Robots.txt is per-domain, which means your staging site and your production site each need their own file. This is easy to forget, and forgetting it has real consequences. Staging sites hold test data, unfinished pages, draft content, and sometimes copies of production data. If a staging domain is crawlable, that material can end up indexed by search engines or quoted in AI answers. Once it is out there, pulling it back is slow and unreliable.
The common pattern is simple. On staging and development domains, disallow everything. A file with a single rule blocking all bots from all paths is enough. It tells every polite crawler to stay away, and it protects unfinished work without any fine-grained decisions. On production, use the selective file from the sample above: allow search bots, block training bots, and keep bots out of low-value URL patterns.
If you serve robots.txt through a URL route, switching between these two files is straightforward: the route can return the strict staging file on staging domains and the selective production file on the production domain. If you serve a single static file, you need a deploy step that swaps the file per environment, and you need to test each domain after deploy. Either way, test both domains. A staging file accidentally deployed to production blocks your search traffic. A production file accidentally deployed to staging leaks your test data. Both mistakes come from using one file for both places.
Django-specific cautions that matter
First, never use robots.txt as access control for admin or private areas. The admin path in the sample is there to save crawl budget, not to protect anything. Robots.txt is a public file. Anyone can read it, and it lists the very paths you want hidden. Authenticated areas need real access control: login requirements, permissions, and proper session handling. If a page should never be seen by an unauthenticated visitor, a robots.txt rule is not protection. It is a signpost pointing at the door.
Second, watch dynamic query-string URLs. Django sites with filters, sorting, and site search generate many URLs that differ only by query parameters. Each one looks like a new page to a crawler. Left alone, bots can spend their crawl budget on thousands of near-identical filter combinations instead of your real content pages. Disallowing the filter and sort parameters for bots keeps the crawl focused and reduces server load. It does not affect human visitors, who can still use filters normally.
Third, treat every deploy as a chance for the file to break. Collecting static files, changing server settings, moving to a new host, or adding a content delivery network can all change what is served at the root URL. Build the live-URL check into your deploy routine. If your site sits behind a CDN or edge proxy, you can also set the rule at the edge in plain terms: the edge layer that sits in front of your Django app can serve the robots.txt response itself, or apply blocking rules before requests reach your app. Edge rules are useful because they act before traffic touches your servers, but they still need the same testing: check the live root URL and check the logs.
Fourth, remember that robots.txt only works if crawlers read it. Most major AI crawlers say they do. Some scrapers do not. If your logs show a blocked token still fetching pages aggressively after your file has been live for a reasonable period, that crawler is either slow to pick up the change or ignoring the file. In that case, robots.txt has done all it can, and you need to look at edge blocking, rate limiting, or access controls for that traffic. Do not keep editing the file hoping a new spelling will fix a crawler that is not reading it.
Common mistakes on Django sites
- Serving the file from a static path instead of the root. The file loads, so it looks done, but crawlers never see it. Always test yourdomain.com/robots.txt, not the static path.
- Blocking search bots by accident. A broad block-all rule copied from a staging file, or a wildcard rule placed above the specific rules, can wipe out search visibility. Keep staging and production files separate and read the final file before deploy.
- Confusing training bots with search bots. Blocking GPTBot and OAI-SearchBot together removes both training use and ChatGPT citations. Decide which one you actually want before you block either.
- Misspelling a bot token. Robots.txt matching is exact for token names. A small typo means that bot is not blocked at all. Copy tokens from official documentation, not from memory.
- Using robots.txt to hide private pages. The file is public. Private and admin areas need login protection, not a disallow rule.
- Forgetting to re-test after deploy. Static file collection and pipeline changes can silently replace the file. A ten-second check of the live URL after each deploy prevents weeks of unprotected crawling.
- Assuming the file works without checking logs. If you never look at server logs, you never learn whether bots stopped. Logs turn a guess into a verified result.
How to confirm it is actually working
Give the change a short settling period, then check three things. First, open the live robots.txt URL and confirm the content matches what you intended to deploy. Second, run the URL through a robots.txt checker and confirm the rules parse correctly for the bots you care about. Third, search your server logs for the blocked tokens over the days after the change and compare the volume with the days before. A sharp drop in page fetches from blocked tokens is the signal you want. A flat line means something is wrong: the file is misplaced, a token is misspelled, or the crawler is ignoring the file.
Also check the other direction. Make sure search bots can still reach your important pages. If your search traffic drops after a robots.txt change, the first suspect should be the file itself. Read it line by line, check for a wildcard rule that is broader than you intended, and confirm the staging file did not reach production. Most robots.txt accidents are found this way, in minutes, once someone actually reads the live file.
The goal is not to block everything. The goal is a file you can explain line by line: these bots are allowed because they bring readers, these bots are blocked because they train models on your work, and these URL patterns are skipped because they waste crawl budget. If you cannot explain a rule, remove it until you can.
Frequently asked questions
Does Django have a built-in setting to block AI crawlers?
No. There is no single Django setting for this. You serve a robots.txt file from the domain root, either as a static file placed so it ends up at the root or through a URL route that returns the file's text for the /robots.txt path. Django handles the delivery. The file handles the policy.
Will blocking GPTBot remove my site from ChatGPT answers?
Not on its own. GPTBot is the training crawler, while OAI-SearchBot handles search and citations. Blocking GPTBot opts you out of training use as described by OpenAI, but citations can continue through OAI-SearchBot unless you block that token too. Decide separately whether you want citations, training use, neither, or both.
Why is my robots.txt file not working on Django?
The most common reason is placement. The file is reachable at a static files path but not at the domain root, so crawlers never read it. The next most common reasons are a misspelled bot token, a staging file deployed to production or the reverse, and a deploy pipeline that overwrote the file. Check the live root URL first, then the token spellings, then your server logs.
Should I block all AI bots, including search and citation bots?
Only if that matches your goal. Blocking everything is simple, but it also removes citations and AI search visibility that some sites value. A selective file that blocks training bots while allowing search and citation bots gives you visibility without training use. Choose based on what your site gains from each type of traffic, not based on a copied block list.
Is robots.txt enough to protect private or admin pages?
No. Robots.txt is a public request, not security. Admin pages, private accounts, and internal tools need login requirements and proper permissions. Use robots.txt to guide polite crawlers on public pages, and use real access control for anything that must stay private.
