Yes, you can block AI crawlers on a Hugo site, and the method is simple: add a robots.txt file that Hugo copies to the root of your published site, with rules that name the AI bots you want to keep out. Hugo builds plain static files, so there is no plugin to install and no code running on each request. Your crawler control lives in files that end up in your output folder, and your host serves them like any other page. This guide walks through the full setup, from a basic robots.txt to per-section rules, host-level enforcement, and testing.
Why Hugo sites need a file-based approach
On a WordPress site, you might install a plugin that edits robots.txt for you. Hugo works differently: when you run a build, Hugo turns your Markdown files and templates into a folder of static HTML, CSS, images, and other files, and that folder is what your host publishes.
This is good news for crawler control. There is no database, no server-side code, and no plugin layer to go wrong. Anything you place in Hugo's static folder is copied to the output folder unchanged, so a robots.txt file placed there appears at the root of your live site, exactly where crawlers look for it.
The trade-off is that robots.txt is your main tool: a static site cannot inspect each request and decide on the fly whether to serve it. For stronger enforcement, pair robots.txt with rules at your host, covered later in this guide.
Step 1: Add a robots.txt through the static folder
The simplest method works for almost every Hugo site. Create a file named robots.txt inside your project's static folder, then rebuild and deploy. Hugo copies it to the root of the output folder, so it becomes available at yourdomain.com/robots.txt.
Here is a starting point that blocks the main AI training crawlers while leaving normal search engines alone:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /Each block names one crawler and tells it not to fetch any path on your site. The final wildcard block allows everyone else, including Googlebot and Bingbot, to crawl normally.
Note that a wildcard rule is only a fallback: if a bot has its own named block, that block wins, so GPTBot follows the GPTBot block even though the wildcard block says Allow.
If you would rather not write the file by hand, a robots.txt generator can build the same stanzas for you. Paste the result into your static folder and deploy as usual.
Step 2: Generate robots.txt with a template instead
The static folder method is enough for most sites, but a second option exists. Hugo supports custom output formats, which means you can define robots.txt as an output of your site and render it from a template, just like a page. The template can then pull values from your site configuration, so the same robots.txt can behave differently across environments or include your sitemap address automatically.
This approach is more work, and for a single rule set it adds complexity without much payoff. It earns its place when you manage several Hugo sites from one setup, or want robots.txt to stay in sync with your site configuration. For most Hugo sites, the static file is the right choice: obvious, easy to review, and impossible to break with a template error.
GPTBot vs OAI-SearchBot: training crawlers and answer engines are different
OpenAI runs more than one crawler, and mixing them up is the most common mistake in AI crawler blocking. Blocking the wrong one can remove your site from ChatGPT answers while leaving training access open.
GPTBot is OpenAI's crawler for training. It collects publicly available pages that may be used to train future models. If your concern is that your writing ends up in training data, GPTBot is the crawler to block.
OAI-SearchBot is a separate crawler that fetches pages so ChatGPT search can cite them in answers. Blocking it does not affect training, and allowing it does not grant training access. It controls whether your pages appear as cited sources in ChatGPT search results.
Many site owners want one but not the other: a writer who does not want their work used for training may still want readers to discover their site through ChatGPT answers. In that case, block GPTBot and leave OAI-SearchBot alone:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /The same split exists elsewhere. Google-Extended controls whether Google may use your content for training its AI models, while Googlebot controls ordinary search indexing. Blocking Google-Extended does not remove you from Google Search. Anthropic's ClaudeBot is a training crawler, and Common Crawl's CCBot collects pages for a public web archive that many AI projects draw on. If you are unsure about a bot's purpose, check its official documentation first — for example, Google Search Central for robots.txt rules and the Hugo documentation for how Hugo handles static files.
Per-section control: blocking part of a Hugo site
Blocking your whole site is rarely what you want. More often, you want readers to find your public articles while keeping a private section out of every crawler's reach. With Hugo, you do this with path patterns in robots.txt, because your content sections map to URL paths in a predictable way.
Suppose your site has a private documentation section published under /docs/internal/. One rule keeps all crawlers out of it:
User-agent: *
Disallow: /docs/internal/That trailing slash matters. Disallowing /docs/internal/ blocks everything under that folder while leaving /docs/ and other sections open. A rule without the slash would block any path starting with those characters, which can catch pages you meant to keep public.
Drafts deserve a special note. Hugo does not publish draft content in a normal production build, so drafts should never reach your live site. If one does appear in your output folder, the problem is your build or deploy settings, not robots.txt. Fix the build, and use robots.txt only for content that is genuinely live but should stay out of crawlers.
Front matter also plays a role, but not the one people expect. It controls how a page is built — its title, date, layout, and whether it appears in lists — but it does not, by itself, keep a crawler out of a published page. If a page is in your output folder and linked from your site, crawlers can find it unless robots.txt or a host rule says otherwise. Treat front matter as a publishing control and robots.txt as a crawler control.
Finally, remember that robots.txt rules are public. Anyone can open yourdomain.com/robots.txt and read it, so a Disallow rule also advertises the path it protects. Never use robots.txt as your only protection for genuinely sensitive content. If a section must not be seen at all, do not publish it to a public host.
Host-level enforcement on Netlify, Cloudflare Pages, and Vercel
Robots.txt is a request, not a lock. Respectful crawlers obey it, but nothing forces a scraper to do so. Hugo hosts give you a second layer: rules that run before a request reaches your files.
On Netlify, you can add redirect and header rules through a configuration file in your project, or through the dashboard, that send a forbidden response to requests whose user-agent string matches a bot you have listed. In plain terms: if an incoming request identifies itself as one of the named AI crawlers, the host answers with a blocked response instead of the page. Because this runs at the edge, it works even for crawlers that ignore robots.txt.
Cloudflare Pages sites behind Cloudflare can use the firewall or managed rules in the Cloudflare dashboard: create a rule matching a known AI bot's user-agent string, or use Cloudflare's built-in managed rule set for AI crawlers, and choose to block or challenge matching requests. No code in your Hugo project is needed; the rule lives in your Cloudflare account and applies before requests reach your static files.
Vercel offers a similar path through its firewall options, where rules matching bot user-agents block them at the edge. As with the other hosts, the rule is defined by its effect: matching requests from named crawlers receive a blocked response.
Testing your robots.txt and verifying it works
A robots.txt file that looks right can still fail quietly: a typo in a bot name, a missing slash, or a deploy that never copied the file to the output folder. Test before you trust it.
- Open the live file. Visit yourdomain.com/robots.txt in a browser and confirm you see your new rules, not an old cached copy. If you see stale content, redeploy and check again.
- Run a tester. Use a robots.txt tester to check specific URLs against specific user-agents. Enter GPTBot and a page URL, and confirm the tester reports it as blocked. Repeat for a public page you want indexed, and confirm it reports allowed.
- Check your server logs. After a few days, look at your host's request logs for the bot names you blocked. A well-behaved crawler should stop requesting blocked paths; requests that continue mean the bot ignores robots.txt, which is exactly what host-level rules are for.
- Confirm search indexing is intact. Search for a few of your public pages and make sure they still appear. Blocking AI training crawlers should not change your Google or Bing presence, and a quick check catches a miswritten rule before it costs traffic.
What robots.txt cannot do
Being direct about the limits matters, because overselling robots.txt leads to false confidence.
First, robots.txt is voluntary. It works because major crawlers agree to follow it. A scraper that ignores the standard will fetch your pages regardless of what your file says. Host-level and firewall rules are the answer for those visitors, not a stricter robots.txt.
Second, robots.txt does not remove content that is already out there. If a crawler already fetched your pages, blocking it today does not delete what it collected yesterday. Likewise, a Disallow rule stops future crawling, but removing a page from an index is a separate step handled through the search engine's own tools.
Third, robots.txt does not hide anything. The file is public by design, and a Disallow entry points directly at the path it protects. Treat anything truly private as unpublished, not merely disallowed.
Fourth, bot names change. New AI crawlers appear regularly, and existing ones get renamed or split into separate training and search crawlers, as OpenAI did. Review your block list a few times a year; a block list from last year will miss this year's crawlers.
Frequently asked questions
Do I need a plugin to block AI crawlers on Hugo?
No. Hugo has no plugin system for this because it needs none. A robots.txt file in your static folder, copied to your output folder on build, does the job for every respectful crawler. Host-level rules add enforcement for the rest.
Will blocking GPTBot remove my site from ChatGPT answers?
Not by itself. GPTBot is the training crawler; OAI-SearchBot is the crawler that fetches pages for ChatGPT search citations. Block GPTBot to opt out of training use, and leave OAI-SearchBot allowed if you still want your pages cited in answers.
Does blocking Google-Extended hurt my Google Search rankings?
No. Google-Extended controls AI training use, while Googlebot controls search indexing. Blocking Google-Extended does not remove your pages from Google Search results.
Can I block AI crawlers from one Hugo section only?
Yes. Add a Disallow rule for that section's path, such as /docs/internal/, while leaving the rest of the site allowed. Watch your trailing slashes, and remember the rule is public, so it is not suitable for truly sensitive content.
How do I know if a bot is actually ignoring my robots.txt?
Check your host's request logs a few days after deploying your rules. If a blocked bot keeps requesting disallowed paths, it is ignoring the file, and you need a host-level or firewall rule to stop it at the edge.
A simple checklist to finish with
Start with the static folder method: a robots.txt file that blocks GPTBot, ClaudeBot, CCBot, and Google-Extended, while allowing ordinary search engines and the answer-engine crawlers you want. Keep per-section rules narrow and check your trailing slashes. Add host-level rules on Netlify, Cloudflare Pages, or Vercel if your logs show crawlers ignoring robots.txt. Test with a robots.txt tester, verify in your logs, and confirm your public pages still appear in search. Pair your robots.txt with a sitemap so the crawlers you do welcome can find your content — a Sitemap generator can build one for you.
