Robots.txt asks AI crawlers not to visit. A Cloudflare Worker stops them at the door. A Worker is a small piece of JavaScript that runs on Cloudflare's edge, in front of your website, and it can inspect every incoming request and decide whether to let it through. If the visitor is GPTBot, ClaudeBot, or any other training crawler you name, the Worker returns a 403 before the request ever reaches your server. It works on any site behind Cloudflare, costs nothing on the free plan for most small sites, and takes about 20 minutes to set up. This guide walks through the whole thing.
The difference between a Worker and robots.txt is enforcement. Robots.txt is a voluntary protocol. A crawler reads it, decides whether to comply, and then does whatever it wants. The major AI companies mostly comply, but not every bot does, and a bot that ignores your file faces no technical barrier. A Worker is a barrier. It does not ask. It checks the user-agent string on every request and refuses the ones on your list. A bot can still lie about its user-agent, and a determined scraper can rotate through residential proxies, but for the named AI crawlers that identify themselves honestly, a Worker is the strongest control a site owner has short of a paid bot management product.
How a blocking Worker works
The flow is simple. A request arrives at Cloudflare for your domain. Before Cloudflare forwards it to your origin server, the Worker runs. It reads the user-agent header, compares it against your list of blocked bot names, and makes one of two decisions. If the user-agent matches a blocked bot, the Worker returns a 403 Forbidden response immediately, and your origin never sees the request. If it does not match, the Worker passes the request through to your site untouched.
That last point matters. A Worker that blocks the wrong user-agent can take your site offline for real visitors. The list must contain bot names only, and you must test it before you deploy it to production traffic. Googlebot, Bingbot, and the AI search crawlers that feed citations are not in the default block list in this guide, and that is deliberate. Blocking them is a separate decision with real traffic consequences.
Step-by-step: set up your first blocking Worker
Step 1: Put your site behind Cloudflare. A Worker only runs on traffic that passes through Cloudflare. If your domain already uses Cloudflare, skip this step. If it does not, create a free Cloudflare account, add your domain, and change your nameservers at your registrar to the ones Cloudflare gives you. DNS changes usually take effect within a few hours. Your site keeps working during the switch, because Cloudflare copies your existing DNS records automatically.
Step 2: Create the Worker. In the Cloudflare dashboard, open Workers and Pages and click Create. Choose the Hello World starter template. You will replace its code with the blocking script in the next step, so do not worry about what the template does.
Step 3: Paste the blocking code. Replace the template with the script below. It blocks the training crawlers that identify themselves by name, and it lets everything else through:
const BLOCKED_BOTS = [
'GPTBot',
'ClaudeBot',
'CCBot',
'Bytespider',
'Meta-ExternalAgent',
'cohere-ai',
'Diffbot',
'anthropic-ai',
'DuckAssistBot',
'Applebot',
];
export default {
async fetch(request) {
const ua = request.headers.get('user-agent') || '';
for (const bot of BLOCKED_BOTS) {
if (ua.includes(bot)) {
return new Response('Access denied.', { status: 403 });
}
}
return fetch(request);
}
};Two notes on this list. First, it blocks Applebot, which is Apple's general crawler, because Apple trains on content Applebot collects unless you opt out, and the opt-out token Applebot-Extended is a control token that never makes requests, so listing it in a user-agent check does nothing. If you want to stay in Apple search results while opting out of training, that requires the robots.txt route, not this Worker. Second, PerplexityBot is not in the default list. PerplexityBot is a search crawler that feeds citations with referral traffic. Add it only if you have decided to stay out of Perplexity answers entirely.
Step 4: Deploy and add a route. Click Deploy. Then go to the Worker's Settings, open Triggers, and add a route for your domain, for example yourdomain.com/*, so the Worker runs on all of your site's traffic. A Worker with no route runs nowhere, which is the most common setup mistake.
Step 5: Test it. Test with a fake user-agent before you trust it. From a terminal, run a request with a blocked bot's user-agent string and confirm you get a 403. Then run the same request with a normal browser user-agent and confirm you get your site. If both behave as expected, the Worker is live. Cloudflare's analytics will show blocked requests within minutes, so you can watch the first real bots hit the wall.
Which bots to block, and which to leave alone
The list above is a starting point, not a religion. Every bot on it is a training or scraping crawler that identifies itself honestly. But your site is not every site, and the right list depends on what you want from AI search.
Block the trainers if you do not want your content in model training data: GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent, cohere-ai, Diffbot, and anthropic-ai. These collect content that may be used to train models, and blocking them costs you nothing in search rankings or AI citations, because none of them feed a search index or an answer engine.
Leave the answer-engine crawlers alone unless you have decided to opt out of AI answers entirely: OAI-SearchBot, Claude-SearchBot, PerplexityBot, and DuckAssistBot. These fetch pages to ground citations in ChatGPT, Claude, Perplexity, and DuckDuckGo AI answers, and they send referral traffic when they cite you. Blocking them removes you from those answers. That is a legitimate choice, but it is a visibility decision, not a training decision, and it should be made deliberately. Our complete list of AI crawlers explains what each bot does so you can decide bot by bot.
Never block Googlebot or Bingbot with a Worker. That is not an AI crawler decision, it is a decision to leave Google and Bing search entirely. If you ever need to block a search crawler, do it in robots.txt where the mistake is easy to spot and reverse, not in edge code.
Costs, limits, and the fine print
The Cloudflare Workers free plan includes 100,000 requests per day, which covers most small and medium sites. If your site gets more traffic than that, the paid plan starts at $5 per month and includes 10 million requests. The Worker in this guide uses almost no CPU time, since it only reads a header and compares strings, so you will not hit the CPU limits that apply to heavier Workers.
There are two limits worth knowing. First, a Worker adds a small amount of latency to every request, usually a few milliseconds, because it runs before your origin responds. For most sites that is invisible. Second, Workers run on Cloudflare's network, which means this technique only works for domains routed through Cloudflare. If your site is not on Cloudflare, the Worker cannot see its traffic at all.
Also remember that a user-agent check only catches honest bots. A scraper that spoofs Googlebot's user-agent will sail through this Worker, and a scraper that rotates IPs and user-agents will not be stopped by a name list. For those, you need verification, not just matching. Our guide to verifying AI bots with reverse DNS explains how to tell a real GPTBot from an impostor, and Cloudflare's own Bot Management product handles the harder cases at a price.
Alternatives inside Cloudflare
A Worker is the most flexible option, but it is not the only one Cloudflare offers, and for many site owners a simpler control is enough.
Block AI Scrapers and Crawlers. Cloudflare's one-click control, under Security then Bots in the dashboard, blocks the known AI scrapers and crawlers that Cloudflare tracks. It is available on the free plan, it needs no code, and it is the right choice if you want a single switch rather than a custom list. See our Cloudflare AI bot blocking guide for the details.
WAF custom rules. A WAF custom rule can match the user-agent string against a list of bot names and return a block, exactly like the Worker, but configured in the dashboard instead of in code. Custom rules are easier to edit than a Worker, and they are the better choice if you expect to change the list often. The Worker is the better choice if you want logic that a rule cannot express, such as different behavior for different paths or time-based rules.
Bot Fight Mode. The free Bot Fight Mode blocks known bad bots and challenges suspicious ones. It is a blunt instrument that can cause false positives for unusual but legitimate visitors, so test it before leaving it on. It is not AI-specific, and it does not replace a targeted AI crawler policy.
Cloudflare also grades its Verified bots on whether they obey robots.txt, a system it announced in October 2026. Keeping your robots.txt consistent and deliberate matters even if you block at the edge, because your robots.txt is now an input to a bot's trust score. Our coverage of Cloudflare's verified bot trust scores explains how that system works.
Workers plus robots.txt: the layered setup
The strongest setup uses both. Keep your robots.txt as the public, standards-based declaration of your policy, and use the Worker as the enforcement layer for the bots that ignore it. The robots.txt tells honest crawlers to stay away and gives you the regulatory backing that the EDPB's 2026 guidance now attaches to it. The Worker catches the crawlers that read your robots.txt and visit anyway, and it stops them before they consume your bandwidth.
This layered approach also protects you from your own mistakes. If the Worker has a bug, your robots.txt is still there asking the honest bots to stay away. If your robots.txt has a typo, the Worker is still blocking the named bots. Redundancy is a feature when the cost of a mistake is your content in a training set or your site invisible in search.
One last check after setup: confirm your robots.txt itself is still reachable. A Worker with a route of yourdomain.com/* runs on every request, including requests for /robots.txt, and a bug in the matching logic could block the file that tells crawlers your policy. Test the robots.txt URL with a normal browser user-agent after deploying, and use our free robots.txt tester to confirm each crawler group still behaves the way you intended.
FAQ
Do I need Cloudflare to use a Worker?
Yes. Workers run on Cloudflare's edge network, so they only see traffic for domains routed through Cloudflare. If your site is not on Cloudflare, add it on the free plan first, then create the Worker.
How much does a blocking Worker cost?
Nothing for most sites. The free plan includes 100,000 requests per day. The paid plan starts at $5 per month for 10 million requests. A blocking Worker uses almost no CPU time, so you are unlikely to hit any other limit.
Will a Worker block Googlebot?
Only if you put Googlebot in the block list, which you should never do. The script in this guide blocks training crawlers only. Always test with a normal browser user-agent after deploying to confirm real visitors and search crawlers still get through.
Is a Worker better than robots.txt?
It is stronger, not better. Robots.txt is a public, standards-based declaration that honest crawlers and regulators respect. A Worker is enforcement for the crawlers that ignore it. Use both: robots.txt for the policy, the Worker for the enforcement.
Can a Worker stop bots that fake their user-agent?
No. A user-agent check only catches bots that identify themselves honestly. A scraper spoofing Googlebot will pass through. For those, verify crawlers with reverse DNS checks against the operator's published IP ranges, or use Cloudflare's Bot Management.
What happens if my Worker has a bug?
A bug in the matching logic can block real visitors, so test before deploying to production traffic. Test with a blocked user-agent and a normal browser user-agent, check that /robots.txt still loads, and watch Cloudflare analytics for the first hour after deployment.
If you prefer the no-code route, start with Cloudflare's one-click control in our Cloudflare AI bot blocking guide, and generate your robots.txt with the free robots.txt generator so the policy layer is in place before you add enforcement.
