If your site runs on Netlify, you have three solid layers for keeping AI crawlers out: a robots.txt file served from your site root, Netlify's own User Agent Blocker extension, and an edge function that refuses requests from known AI crawlers before they ever reach your pages. Used together, they cover the polite crawlers, the lazy crawlers, and the crawlers that ignore robots.txt entirely. This guide walks through all three, in order, so you can stop at the layer that fits your comfort level.
The short answer: start with robots.txt, because every reputable crawler reads it. Then turn on the User Agent Blocker extension if you want a no-code block that runs at the edge. Then, only if you need to stop crawlers that ignore robots.txt, add an edge function that checks each incoming request's user-agent header against your list of known AI crawlers and returns a refusal response to matches. Each layer is independent, so you can add them one at a time and verify each one before moving on.
Why Netlify Needs a Different Approach
Netlify is not a traditional server with a configuration file you can edit for bot blocking. There is no server-level access file to drop rules into, and you cannot install a plugin the way you would on WordPress. What Netlify does give you is a modern edge: your site is served from a global content delivery network, and you can run small functions at that edge, close to the visitor, before your pages are served. Bot control on Netlify therefore means working with the edge, not against it.
That architecture has a real advantage. An edge function runs before your page content is generated or served, so a blocked crawler never triggers a full page render, never consumes a serverless invocation for your content, and never shows up in your analytics as a visit. It also has a limitation worth knowing: edge functions run on every request they are attached to, so a badly written one adds latency to real visitors too. Keep the check simple — a list comparison against the user-agent header — and the cost is negligible. Do not make the edge function fetch a database or call an external service on every request; that turns a guard into a bottleneck.
Layer 1: robots.txt at Your Site Root
Every Netlify site can serve a robots.txt file, and this is where to start. Place a file named robots.txt in your site's publish directory — the folder that gets deployed as your site root, which for many frameworks is called public, dist, or out, depending on your build setup. After your next deploy, the file should be available at yourdomain.com/robots.txt. If it is not, the file is in the wrong folder or your build is not copying it; check your deploy log for the publish directory name before changing anything else.
The content follows the standard pattern used across this blog. Give each AI training crawler its own group with a disallow for the whole site, and leave the search crawlers you want — Googlebot, Bingbot, and the answer-engine crawlers that bring citations — out of the block list. The key tokens to know: GPTBot for OpenAI training, ClaudeBot for Anthropic training, CCBot for Common Crawl, Google-Extended as the token that controls Gemini training use, and Applebot-Extended as the equivalent token for Apple. Keep Googlebot itself allowed if you want search traffic; blocking the training token does not block search, and confusing the two is one of the most common and costly mistakes in AI crawler control.
After deploying, verify the live file by visiting yourdomain.com/robots.txt in a browser, and then test a few URLs against it with the free robots.txt tester. If you do not want to write the file by hand, the free robots.txt generator builds named groups for the major AI crawlers and produces a file you can drop straight into your publish directory.
Layer 2: Netlify's User Agent Blocker Extension
Netlify's own documentation describes a User Agent Blocker extension that blocks web requests from common AI crawlers, search crawlers, and SEO engines using a preset list. It works through an edge function under the hood, but you do not write that function yourself: you install the extension, choose which categories of user agents to block, and redeploy.
The setup path, as documented by Netlify, runs through your project dashboard: open Extensions in the sidebar, find User Agent Blocker, install it for your team, then activate it for one project at a time and select the traffic you want to block, including the AI crawler options. After saving, redeploy the site so the edge function takes effect. The extension's list covers common AI crawlers, but Netlify itself notes that the list does not include every possible AI crawler or bot, so treat it as a strong baseline rather than a complete solution.
This layer is the right choice for most Netlify site owners. It requires no code, it runs at the edge before your content is served, and it can be changed from the dashboard without a code deploy beyond the redeploy that activates it. Its limits are also clear: you are choosing from Netlify's preset list, you cannot add a brand-new crawler the day it appears, and you cannot write custom logic, such as allowing one crawler on your blog but blocking it on your pricing pages. If either of those matters to you, the next layer is the answer.
Layer 3: A Custom Edge Function for Full Control
A custom edge function gives you complete control over which requests are refused. Netlify's developer documentation walks through exactly this pattern for blocking AI bots: an edge function placed in the project's edge-functions folder inspects the user-agent header of each incoming request, compares it against a list of known AI crawler names, and returns a refusal response — Netlify's own example uses a 401 status — for matches, while letting all other requests continue to the site normally. You configure the function to run on the paths you choose, typically every path, through your site's configuration file.
Building your own version follows the same shape, described here as a walkthrough rather than a code sample so you can adapt it to your framework. First, keep a list of the crawler tokens you want to block, using lowercase comparison so that a crawler that varies its capitalization still matches. Second, in the edge function, read the user-agent header from the incoming request and check whether it contains any token from your list. Substring matching is deliberate here: crawler user agents often carry version numbers and extra details around the token, so an exact match would miss most of them. Third, if there is a match, return a short refusal response immediately, without fetching the page. If there is no match, pass the request through to your site untouched.
There are three details that separate a good edge function from a troublesome one. First, never block by a fragment that also appears in real browser user agents. Short, generic words are dangerous; full crawler tokens are safe. Second, keep your allowlist in mind: search crawlers, social preview fetchers, and uptime monitors all send user agents too, and a careless list can block the tools that keep your site visible and monitored. Test with a search crawler's user agent and a link preview fetcher before you consider the function done. Third, log what you block, at least at first. A simple log line per blocked request, reviewed for a week, tells you whether you are catching real crawlers or accidentally catching something you value. Most edge platforms let you view function logs from the dashboard without extra setup.
Which Layer Should You Use?
Use robots.txt alone if your concern is reputable crawlers and you want the lightest possible setup. It costs nothing, it is understood by every search engine, and it is the mechanism AI companies document for opting out. Add the User Agent Blocker extension if you want enforcement against crawlers that ignore robots.txt but you do not want to write code. Add a custom edge function if you need a list Netlify's preset does not cover, if you need different rules for different sections of your site, or if you want to log exactly what you are blocking.
Using all three together is not overkill; it is defense in depth, and each layer covers a failure mode of the others. Robots.txt covers the crawlers that read it. The extension covers the common crawlers that do not. The custom function covers the new or niche crawler that neither of the first two knows about yet. The cost of running all three is one extra file, one dashboard toggle, and one small function — a reasonable price for a site whose content is its main asset.
Testing Your Netlify Bot Blocks
Testing is where most Netlify bot setups fail, because each layer is invisible when it works. Test each layer separately. For robots.txt, fetch yourdomain.com/robots.txt after a deploy and check it against a tester with several crawler names. For the extension and the edge function, you need to simulate a crawler request, because a normal browser visit will pass through untouched — which is exactly the point. The simplest check is a command-line request that sets a crawler user agent and shows the response status: a blocked request should return a refusal status, while the same URL with a normal browser user agent should return your page. If both return the page, the edge layer is not running; check that the site was redeployed after the change and that the function is attached to the right paths.
Also test the paths you care about most: your homepage, a blog article, and any page that drives revenue or signups. A block that works on the homepage but not on article pages, because the function is attached to only one path pattern, leaves your most valuable content exposed. And test a social preview: share a link in a messaging app and confirm the preview still renders, which confirms you have not blocked the preview fetchers alongside the AI crawlers.
Common Netlify Mistakes
Putting robots.txt in the source folder instead of the publish folder
Netlify deploys the publish directory, not your whole repository. A robots.txt file sitting in your project root is not deployed unless your build copies it into the publish directory. This is the single most common reason a Netlify robots.txt "does not work." Check your deployed files, not your repository, to confirm what is live.
Forgetting to redeploy after a dashboard change
Extension changes and edge function changes take effect on redeploy. If you change a setting and then test the live site without redeploying, you are testing the old configuration and will conclude, wrongly, that the change did nothing. Make redeploying part of the change, every time.
Blocking preview and monitoring bots by accident
Social link previews, search console fetchers, and uptime monitors are all bots in the technical sense, and a broad block list can catch them. If your link previews suddenly go blank or your uptime monitor reports your site as down while it loads fine in a browser, your block list is the first suspect. Keep a short allowlist of the fetchers you rely on, and check it before expanding your block list.
Assuming one layer is enough
Robots.txt alone does not stop a scraper that ignores it. An edge function alone does not tell reputable crawlers your policy, which matters if you ever want to show that you opted out properly. The layers are complementary, and the strongest setup uses at least the first two.
Frequently Asked Questions
Can I block AI crawlers on Netlify without writing code?
Yes. A robots.txt file in your publish directory covers reputable crawlers, and Netlify's User Agent Blocker extension adds edge-level blocking from the dashboard without code. A custom edge function is only needed for custom lists or per-section rules.
Does Netlify support the traditional server access file for bot blocking?
No. Netlify does not use that server configuration approach. Bot blocking on Netlify is done through robots.txt, the User Agent Blocker extension, or edge functions, as described in Netlify's own developer documentation.
Will blocking AI crawlers hurt my Google search rankings?
Not if you block the right tokens. Blocking training tokens such as GPTBot, ClaudeBot, and Google-Extended does not block Googlebot, which is the crawler behind search rankings. Keep Googlebot allowed, and your search visibility is unaffected by training opt-outs.
How do I know my edge function is actually running?
Send a request with a blocked crawler's user agent and confirm you get a refusal status, then send the same request with a browser user agent and confirm you get the page. Also check your function logs in the Netlify dashboard for blocked requests after the first day.
Should I block every AI crawler, or only the training ones?
That depends on whether you want AI citations. Training crawlers feed model training; search and answer crawlers feed citations that can send you traffic. Many site owners block the training tokens and allow the answer crawlers. Decide based on your own traffic, and keep the two groups separate in both your robots.txt and your edge function list.
What happens when a new AI crawler appears?
Add its token to your robots.txt and to your edge function list, then redeploy. The preset extension list will eventually catch common new crawlers, but your own list is the fastest way to cover a crawler you have spotted in your own logs.
Netlify gives you a clean path from a simple text file to full edge enforcement, and you can walk it one layer at a time. Start with robots.txt today, add the extension when you want enforcement without code, and add a custom edge function when you need a list of your own. Verify each layer with a real crawler-style request, and your content will be protected by policy and by enforcement, not by hope.
