AI crawlers can quietly eat a large share of your server's bandwidth. They fetch your pages by the thousands, often hitting filtered, sorted, and paginated URLs that no human ever visits, and they do it around the clock. The fix is a measurement habit, not a guess: check your logs to see which bots visit and how much they take, then cut the waste with robots.txt rules, crawl-delay where it works, and edge controls for the bots that ignore the rules. This guide walks through all three steps.
Most site owners discover the problem the same way. Hosting costs creep up, pages feel slower at odd hours, or the analytics show traffic that does not behave like people. Then someone opens the server logs and finds a single AI crawler that requested 40,000 URLs in a day, most of them search-result pages and calendar archives that should never have been crawled at all. Training crawlers do not browse your site the way a reader does. They follow every link, including the ones your own navigation hides, and they keep going until they have seen everything.
Step 1: Measure how much AI crawlers actually take
Start with the numbers you already have. If you use Cloudflare, open the analytics dashboard and look at the bot traffic breakdown. Cloudflare classifies requests by bot type, so you can see AI crawlers separately from search crawlers and from verified good bots. If you use Google Search Console, the crawl stats report shows how many requests Googlebot makes per day and how much data it downloads. That covers Google only, but it gives you a baseline for what normal search crawling looks like on your site.
For the full picture, look at your raw server logs. Every request in the log carries a user-agent string that names the crawler. Search the log for the tokens that matter: GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, CCBot, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider, DuckAssistBot, meta-externalagent, and facebookexternalhit. Count the requests per bot, add up the bytes served, and compare that with your total traffic. Two numbers matter: the share of requests and the share of bandwidth. A bot that makes 5 percent of requests but takes 30 percent of bandwidth is fetching your heaviest pages, probably images and large documents, and that is where the cost lives.
Also look at which URLs each bot requests. A healthy search crawler visits your real pages: posts, product pages, category pages. A training crawler often visits the long tail of your site: every page of a paginated archive, every combination of a filtered search, every dated calendar view. If 80 percent of a bot's requests go to URLs no human would ever open, you have found waste, not value. Write down the top three bots by bandwidth and the top URL patterns they hit. That short list is your entire action plan.
What the public data says about AI crawler volume
Individual logs tell you about your site. Public studies tell you the pattern is not unique to you. Cloudflare, which sits in front of a large share of the web, has talked publicly about AI crawler traffic growing fast enough to change how it prices and polices bot access, and in 2025 it launched a pay-per-crawl system that lets site owners charge AI crawlers for access. That is a strong signal: the largest edge network on the web considers AI crawler volume a real cost for site owners, not a rounding error.
Log studies from independent researchers point the same way. One widely cited study logged more than 1,600 AI-crawler visits over 36 days on a single site and found that the bots requested robots.txt hundreds of times but never once requested llms.txt. Another found that 97 percent of sites with an llms.txt file had never received a single request for it. The point is not the llms.txt numbers. The point is the volume: hundreds of requests per bot per month on an ordinary site, concentrated on robots.txt and then on pages, and that is the polite, well-behaved traffic. Bots that ignore robots.txt do not show up in those neat counts at all.
Do not trust any article that gives you a single global number for AI crawler traffic, including this one. The share varies wildly by site type. A documentation site with clean URLs might see a small share. A forum, a marketplace, or a site with faceted navigation can see AI crawlers take a large share of requests, because those sites have near-infinite URL combinations for a crawler to walk. Your own logs are the only number that matters for your bill.
Step 2: Cut the waste with robots.txt
The cheapest fix is a robots.txt file that does two things: blocks the training crawlers you do not want, and keeps crawlers out of the URL patterns that waste bandwidth. Both are simple text rules.
First, block the training bots while leaving the answer bots alone. The training crawlers are GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, and meta-externalagent. The answer crawlers, OAI-SearchBot, Claude-SearchBot, and PerplexityBot, fetch pages so AI products can cite them, and blocking them removes you from AI answers. If you want citations, leave them allowed. Here is the core of the file:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: meta-externalagent Disallow: / User-agent: * Disallow:
Second, block the low-value URL patterns for every crawler, including the search engines. If your site has faceted search, internal search results, or calendar archives, those URLs multiply your crawlable surface by thousands with no benefit. Add Disallow lines for them under the wildcard group. Common examples are /search/, /*?sort=, /*?filter=, and dated archive paths. Only block patterns you are sure carry no content you want indexed. When in doubt, check your logs first: the URL patterns a bot hits most are usually the ones worth blocking.
Our free robots.txt generator builds this file with checkboxes, and the free robots.txt tester lets you check any URL against any bot before you deploy. Test first, because a typo in this file can block Googlebot from your whole site, and that mistake costs far more than any crawler ever will.
Step 3: Slow down the bots that respect the rules
Robots.txt has a crawl-delay directive that asks a crawler to wait a number of seconds between requests. It looks like this in the file: a Crawl-delay: 10 line inside a user-agent group. The catch is that support is uneven. Bing and some other crawlers honor it. Google ignores it entirely and expects you to manage Googlebot's crawl rate in Google Search Console instead. The major AI crawlers do not document crawl-delay support, so do not rely on it for them.
That does not make crawl-delay useless. It helps with the smaller crawlers that do respect it, and it costs one line to add. Just do not treat it as your main defense against GPTBot or CCBot. For those, the Disallow rules above are the mechanism that works, because the major AI crawlers do respect robots.txt access rules even if they ignore the delay hint.
A better lever for crawl rate is your own site structure. Pagination that exposes thousands of near-identical pages, faceted navigation that generates a new URL for every filter combination, and calendar widgets that link to every month since 2015 are all invitations for a crawler to walk forever. Add nofollow to faceted links, cap your archive pagination, and keep internal search results out of the crawlable web. These changes reduce crawler load for every bot at once, including the ones you want to keep, and they also help human visitors.
Step 4: Enforce against the bots that ignore robots.txt
Some crawlers ignore robots.txt. That is not a theory; it is a documented behavior for a set of aggressive scrapers and unidentified bots that rotate user-agent strings. Robots.txt cannot stop them, because the file is a request and they have chosen not to listen.
Enforcement has to happen at the edge, in front of your server. If you use Cloudflare, its bot management and managed rules can block known AI bots and challenge suspicious traffic before it reaches your origin. That is the simplest enforcement path for most sites, and it is the reason Cloudflare's pay-per-crawl launch matters: the edge is where bot control actually works. If you run your own server, your web server software can inspect the user-agent header and return a 403 response for known bad crawlers. The details depend on your stack, so describe the rule in plain language to your developer or hosting support: block requests whose user-agent matches the list of training crawlers you have already decided to exclude, and log what gets blocked so you can check for false positives.
One caution: do not block by IP address unless you know what you are doing. AI crawlers run on large, shared cloud ranges, and blocking an IP range can take down legitimate services that share it. User-agent matching plus edge rules is safer and easier to reverse.
How much can you actually save?
It depends on how much of your traffic is crawler waste today. On a small blog with a few hundred pages, the savings are modest: a few gigabytes a month and a slightly quieter server. On a large site with faceted navigation, the savings can be dramatic, because the blocked URL patterns are the ones generating the most requests. The honest way to find out is to measure before and after. Record the top bots by requests and bandwidth this week, ship the robots.txt changes, and check the same numbers in two weeks. Crawlers cache robots.txt for up to 24 hours, so give the change a day to propagate before judging it.
Also watch your search traffic, not just your crawler traffic. If organic search visits drop after the change, you blocked something you should not have, most likely Googlebot or a path with real content. Roll back the file and check the rules in a tester. A good robots.txt change reduces crawler requests while search traffic stays flat or grows, because Googlebot spends its crawl budget on your real pages instead of your filtered archives.
Frequently asked questions
How do I know if AI crawlers are costing me money?
Check your server logs or your CDN analytics. Add up the requests and bytes served to AI crawler user-agents like GPTBot, ClaudeBot, CCBot, and Bytespider, and compare with your totals. If a bot's share of bandwidth is much larger than its share of requests, it is fetching heavy pages, and that is where the cost is.
Does blocking GPTBot stop ChatGPT from citing my site?
No. ChatGPT citations come from OAI-SearchBot, a different crawler. Blocking GPTBot stops OpenAI's training crawler. Leave OAI-SearchBot allowed and your pages can still appear in ChatGPT answers.
Is crawl-delay supported by AI crawlers?
Not reliably. Bing and some smaller crawlers honor it, but Google ignores it and the major AI crawlers do not document support for it. Use Disallow rules for AI crawlers and manage Googlebot's rate in Google Search Console instead.
Can I charge AI crawlers for access?
Cloudflare launched a pay-per-crawl system in 2025 that lets site owners using its network set a price for AI crawler access. It is the most practical version of this idea today, but it only applies if your site sits behind Cloudflare. Robots.txt blocking remains the free, universal option on any host.
Will blocking CCBot keep my content out of AI training data?
It helps a lot. Common Crawl publishes its dataset openly and many AI companies train on it, so blocking CCBot cuts off a major pipeline. It does not remove content that was already collected, and it does not stop other crawlers, so block the other training bots in the same file.
What should I do about bots that ignore robots.txt?
Enforce at the edge. Use your CDN or firewall to block or challenge requests with those user-agent strings before they reach your server. Do not rely on IP blocking alone, and keep logging what gets blocked so you can catch false positives.
AI crawler bandwidth is a cost you can measure and a cost you can cut. The pattern is always the same: find the bots in your logs, block the trainers and the junk URL patterns in robots.txt, and enforce at the edge for the bots that ignore the file. Do that, measure again in two weeks, and you will know exactly how much quieter your server got.
