Most site owners think about AI crawlers in terms of copyright and visibility. There is a third cost that shows up quietly on the hosting bill: bandwidth. Every page an AI crawler fetches is a page your server had to build and send, and on a busy site the total can be large enough to notice. This guide shows how to measure what AI crawlers actually cost you, and how to cut that cost without harming the search traffic that pays your bills.
The direct answer: measure first, in your access logs or CDN reports, by counting requests and bytes per crawler user-agent. Then cut the waste in order: block the crawlers you do not want in robots.txt, slow down the ones you keep with crawl-delay where it is honoured, and cache aggressively so repeated fetches cost almost nothing.
Why AI crawler traffic costs real money
A human visitor reads a few pages and leaves. A crawler reads systematically: every new page, every updated page, and sometimes the same page again and again as different systems re-check it. AI crawlers add a multiplier, because the same content may be fetched by a training crawler, a search crawler, and a user-triggered fetching bot from the same company, each on its own schedule.
The cost is not just bandwidth. Each request consumes server CPU to build the page, database queries to fill it, and log storage to record it. On shared hosting, that shows up as slowdowns for real visitors during crawl spikes. On metered hosting or a CDN with overage charges, it shows up as money. On a small site the amounts are trivial; on a site with thousands of pages, or with large images and media files, they are not.
The important point is that this cost is measurable. Guessing whether bots are "a lot" of your traffic is a waste of time when your logs already contain the answer. Measure first, and every decision after that gets easier.
Step 1: Measure bot traffic in your logs
Start with the simplest question: how many requests in the last 30 days came from known AI crawler user-agents, and how many bytes did those requests transfer? Your hosting control panel, your CDN analytics, or your raw access logs can all answer this, depending on what your host provides.
Group the results by user-agent token, not by individual bot names you invent. GPTBot, ClaudeBot, CCBot, PerplexityBot, Applebot, Bytespider, and the search bots such as OAI-SearchBot and Claude-SearchBot are the tokens to look for first. For each one, note three numbers: requests, bytes transferred, and the share of your total traffic. A crawler that makes many tiny requests is a different problem from one that downloads your largest media files, and the fix differs too.
Watch out for fakes. A scraper that pretends to be GPTBot will inflate your GPTBot numbers and lead you to blame the wrong operator. For the biggest surprises in your data, verify a sample of addresses with reverse DNS against the operator's published pattern before acting on the numbers. Decisions based on fake traffic tend to be wrong in both directions: blocking a real crawler you actually want, or ignoring a scraper you should have challenged.
Step 2: Work out what the traffic is worth to you
Not all bot traffic is waste. A search crawler that brings citations and referral visits earns its bandwidth. A training crawler that never sends you a visitor does not. A user-triggered fetching bot that opens your page because a person asked about it sits somewhere in between: it costs you a fetch, but it serves a real person at the end of the chain.
Sort your crawlers into three buckets. Keep: search crawlers whose answers send you traffic, and any crawler whose output you value. Slow down: crawlers you want to keep but that crawl too aggressively. Block: crawlers whose output gives you nothing and whose volume is large. This is a business decision, not a technical one, and two sites with identical logs can reasonably reach different answers. A news site that lives on citations will keep more search bots than a documentation site that gets nothing from them.
As an illustration only: imagine a site serving 100 gigabytes a month, of which logs show 20 gigabytes going to AI crawlers. Cutting the unwanted half of that bot traffic would save 10 gigabytes, which on a metered plan might be a few dollars or a meaningful amount depending on the plan. The numbers in that example are invented to show the arithmetic; your own logs give you the real ones, and they are the only ones that matter.
Step 3: Block the crawlers in the "block" bucket
For well-behaved crawlers, robots.txt is the cheapest fix available. It costs nothing to write, and a crawler that honours it stops fetching before your server does any real work. Add a group per crawler token, disallow the paths you want protected, and keep Googlebot and Bingbot allowed so ordinary search is untouched.
Test the file before deploying, and remember that crawlers cache robots.txt, so a change takes time to show up in your logs. If a blocked crawler's requests do not fall within a few days, check your file for typos, and check whether the requests are really from that crawler or from a faker wearing its name. Real crawlers that ignore robots.txt, and all fakers, need the next step instead.
Step 4: Slow down the crawlers you keep
Some crawlers honour crawl-delay, a robots.txt directive that asks for a pause between requests. It is not part of the formal robots.txt standard, and not every crawler respects it, but for the ones that do, it is a gentle way to spread load without losing the crawler entirely. A delay of a few seconds between requests can turn a spike that slows your site into background noise.
For crawlers that ignore crawl-delay, use rate limiting at the edge: your CDN or hosting firewall can limit how many requests a single user-agent or address range may make in a minute, and challenge or block the excess. Apply limits narrowly — to the specific crawler, not to all traffic — so real visitors never notice. And never rate-limit Googlebot aggressively; a search crawler that cannot fetch your pages on time can hurt your indexing.
Step 5: Make every remaining fetch cheaper
Whatever crawlers remain, make their visits cheap. Cache full pages at the edge so repeat fetches are served from cache instead of rebuilding the page. Set sensible cache lifetimes for images and static files so a crawler that re-checks your media does not re-download it every time. Compress responses, and avoid serving enormous unoptimised images to any visitor, bot or human.
This step helps humans too, which is why it should come before any aggressive blocking. A site that serves cached, compressed pages can absorb far more crawler traffic without slowing down or costing more, and the crawlers you keep will fetch faster and more reliably. Performance work is bot management work, even though nobody calls it that.
Media-heavy sites deserve a special note, because images and video change the arithmetic completely. A text page might transfer a few tens of kilobytes, while a page with several large, unoptimised images can transfer several megabytes per fetch. Multiply that by a crawler that re-checks every page weekly, and your media library becomes the largest line in your bot bandwidth bill. The fix is unglamorous but effective: resize images to the size they are actually displayed at, compress them, and serve modern formats where your audience's browsers support them. Then check whether your crawler traffic is even fetching those images, or whether it is mostly fetching HTML. If a crawler is downloading your full-resolution images for no reason, a robots.txt rule that keeps it away from your media folder can save more bandwidth than blocking it from your article pages ever would, while leaving your text content available for citations. Measure the split between HTML and media bytes before you decide, because the cheapest fix is the one aimed at the largest number.
A monthly routine that keeps costs down
- Once a month, check bot requests and bytes by user-agent in your logs or CDN reports.
- Move any new AI crawler token into the keep, slow-down, or block bucket, and update robots.txt accordingly.
- Verify suspicious spikes with reverse DNS before blaming a real operator.
- Re-test robots.txt after every edit, and confirm Googlebot and Bingbot are still allowed.
- Review cache hit rates: if bots are bypassing your cache, find out why.
Start with a clean robots.txt built by the free robots.txt generator, check it with the robots.txt tester, and read our guide to crawl-delay and which crawlers honour it before you rely on delays to control load.
Frequently asked questions
How do I know how much bandwidth AI crawlers use?
Check your access logs or CDN analytics, grouped by crawler user-agent, for requests and bytes transferred over the last 30 days. That gives you the real number for your site; averages from other sites do not apply to yours.
Will blocking AI crawlers save me money?
It saves the bandwidth and server work those crawlers consumed, which matters on metered hosting, CDN overage plans, or shared hosting where crawl spikes slow real visitors. On unmetered hosting the saving is performance, not money.
Does crawl-delay work for all AI crawlers?
No. It is not part of the formal robots.txt standard, and support varies. Treat it as a request that some crawlers honour, and use edge rate limiting for crawlers that do not.
Can I block bots without hurting Google rankings?
Yes, as long as Googlebot stays allowed and your rules target specific AI tokens. Test the file, and avoid wildcard blocks that catch search crawlers by accident.
What if a blocked crawler keeps visiting?
First check for typos and for faked user-agents, using reverse DNS verification. A genuine crawler that ignores robots.txt needs edge-level rate limiting or blocking, since robots.txt alone cannot stop it.
Is caching enough on its own?
Often it is a large part of the answer. Cached pages cost almost nothing to serve, so even heavy crawler traffic becomes cheap. Combine caching with robots.txt for the best result: block what you do not want, and make what remains inexpensive.
