Next.js sites need AI-crawler controls just like any other stack, but the framework gives you four different places to put them — and only the first one is why so many Next.js robots.txt files silently do nothing. The short answer: use app/robots.ts (or a static public/robots.txt) as your primary control, add middleware user-agent blocking only if you need enforcement rather than requests, and use next.config.js headers for X-Robots-Tag on non-HTML files. This guide shows all four, with working code and the mistakes to avoid.
Method 1: robots.ts — the App Router way (recommended)
If your project uses the App Router, create app/robots.ts. Next.js generates /robots.txt from it at build time:
import type { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
return {
rules: [
// AI training crawlers: blocked
{ userAgent: 'GPTBot', disallow: '/' },
{ userAgent: 'ClaudeBot', disallow: '/' },
{ userAgent: 'CCBot', disallow: '/' },
{ userAgent: 'Bytespider', disallow: '/' },
{ userAgent: 'Google-Extended', disallow: '/' },
// AI search-assistant crawlers: allowed (they cite you)
{ userAgent: 'OAI-SearchBot', allow: '/' },
{ userAgent: 'Claude-SearchBot', allow: '/' },
{ userAgent: 'PerplexityBot', allow: '/' },
// Everyone else: normal rules
{ userAgent: '*', allow: '/', disallow: '/api/' },
],
sitemap: 'https://www.example.com/sitemap.xml',
}
}This is the training-vs-answering split expressed in code: the trainers are out, the bots that cite you in AI answers stay in, and each token is named explicitly so nothing depends on fragile wildcard behavior. Generate the equivalent rules with our robots.txt generator if you'd rather not hand-write them, then paste the tokens into the rules array.
One rule: pick either robots.ts or a static file (Method 2) — not both. If both exist, the static file in public/ takes precedence and your robots.ts is silently ignored, which is exactly the "why isn't my robots.txt updating!" bug that fills forums.
Method 2: public/robots.txt — the simple static file
For the Pages Router, or if you just want the simplest possible setup, put a plain robots.txt in the public/ directory. Next.js serves anything in public/ at the root, so public/robots.txt becomes yoursite.com/robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: *
Disallow: /api/No build step, no TypeScript, and every deployment platform understands it. The trade-off is that it's static: if you want different rules per environment (say, blocking everything on staging), you need build-time logic, which is where robots.ts or middleware earns its keep.
After deploying either method, verify the live file with our robots.txt tester — paste your URL and a bot token like GPTBot and confirm it reports "disallowed." Cached deployments are the usual culprit when the tester disagrees with your code.
Method 3: middleware user-agent blocking — enforcement, not requests
Here is the critical distinction most guides blur: robots.txt is a request, middleware is enforcement. A well-behaved crawler reads robots.txt and obeys it. A scraper that ignores robots.txt will also ignore your robots.txt. If you need to actually refuse requests from known AI-bot user agents — at the cost of running code on every request — middleware is the tool.
Create middleware.ts at your project root:
import { NextResponse } from 'next/server'
import type { NextRequest } from 'next/server'
const AI_BOT_PATTERN = /GPTBot|ClaudeBot|CCBot|Bytespider|PerplexityBot|anthropic-ai|cohere-ai|FacebookBot|meta-externalagent/i
export function middleware(request: NextRequest) {
const ua = request.headers.get('user-agent')
if (ua && AI_BOT_PATTERN.test(ua)) {
return new NextResponse('Forbidden', { status: 403 })
}
return NextResponse.next()
}
export const config = {
matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
}The matcher skips Next.js's own static assets so you're not spending edge compute on requests that were never going to match. Tune the token list to your policy — the example above is aggressive (it includes PerplexityBot, which cites you); remove any token you want to allow.
Three honest caveats about middleware blocking
1. User agents can be spoofed. Middleware sees the User-Agent header the client claims. A scraper that wants your content sends Mozilla/5.0 and walks right past the regex. Middleware stops honest bots and lazy scrapers; it does not stop anyone determined. For high-value content, pair it with a WAF or bot-management service rather than relying on headers alone.
2. It can't verify identity cheaply. Proper bot verification means reverse-DNS lookups (does this IP really belong to OpenAI/Anthropic?). Doing DNS on every request in edge middleware is slow and expensive. Middleware is a coarse filter; treat anything it "verifies" as unverified.
3. It costs you on every request. Middleware runs before every matched request, including real users. On Vercel's Edge Network that's edge compute you're paying for (or burning through included usage). For most sites, robots.txt handles the well-behaved 95% and middleware is only worth it for the abusive remainder — measure before you deploy it site-wide.
Method 4: X-Robots-Tag headers for non-HTML files
Robots meta tags only work in HTML. If your Next.js app serves PDFs, images and other files that crawlers can index directly, control them with the X-Robots-Tag HTTP header in next.config.js:
/** @type {import('next').NextConfig} */
const nextConfig = {
async headers() {
return [
{
source: '/downloads/:path*',
headers: [
{
key: 'X-Robots-Tag',
value: 'noindex, nofollow',
},
],
},
]
},
}
export default nextConfigBe precise about what this does and doesn't do: X-Robots-Tag supports the same directives as robots meta tags — noindex, nofollow, nosnippet and friends. It keeps files out of search indexes (including the indexes behind AI Overviews). It is not an AI-training opt-out: there is no widely honored noai meta directive, and Google's training control remains the Google-Extended robots.txt token. Don't let a header give you a false sense of coverage — pair it with robots.txt rules for the actual AI crawlers.
Putting it together: a sane default stack
For a typical Next.js content or marketing site, this combination covers the bases without over-engineering:
- robots.txt (via
robots.tsorpublic/robots.txt) — block training crawlers, explicitly allow the answering bots, disallow/api/and draft routes for everyone. This is your primary control and the one every honest bot reads. - Middleware — only if logs show bots ignoring robots.txt. Start narrow (one or two abusive tokens), watch for false positives, and remember spoofing limits what it can prove.
- X-Robots-Tag —
noindexon non-HTML assets you don't want indexed or cited, like internal PDFs or staging exports. - Verification — after every deploy, fetch
/robots.txtin an incognito window and run it through the robots.txt tester. Next.js caching (and CDN caching in front of it) is the number-one reason "I updated robots.txt but nothing changed."
Common Next.js mistakes
Blocking /api/ for AI bots but leaving it open in the wildcard group. AI crawlers with no matching group follow *. If your wildcard allows /api/, naming GPTBot to disallow it changes nothing for any bot you didn't name. Write the API disallow in the wildcard group itself.
Forgetting draft/preview routes. Next.js preview mode and draft routes are just URLs. If /drafts/* is reachable, it's crawlable. Disallow it explicitly.
Staging indexed as production. The classic: staging.yoursite.com ships the same robots.ts as production and gets indexed, or worse, outranks production. Use an environment check in robots.ts to disallow everything when process.env.VERCEL_ENV !== 'production':
import type { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
if (process.env.VERCEL_ENV !== 'production') {
return {
rules: [{ userAgent: '*', disallow: '/' }],
}
}
return {
rules: [
{ userAgent: 'GPTBot', disallow: '/' },
{ userAgent: 'ClaudeBot', disallow: '/' },
{ userAgent: '*', allow: '/', disallow: '/api/' },
],
sitemap: 'https://www.example.com/sitemap.xml',
}
}Assuming middleware replaces robots.txt. It doesn't. Honest bots never reach your middleware's decision because they check robots.txt first and never request disallowed paths. Middleware only matters for the dishonest ones. You need both layers, doing different jobs.
Testing your setup end-to-end
Don't trust the code — trust the HTTP responses. After deploying, run these checks from the terminal:
1. Fetch robots.txt exactly as a crawler sees it:
curl -s https://www.example.com/robots.txtCompare the output line-by-line with what you wrote. If it doesn't match, suspect CDN caching (purge it) or the dual-file conflict (a public/robots.txt shadowing your robots.ts).
2. Confirm your middleware actually fires:
curl -s -o /dev/null -w "%{http_code}" -A "GPTBot/1.0" https://www.example.com/blog/hello-world
# expect: 403
curl -s -o /dev/null -w "%{http_code}" -A "Mozilla/5.0" https://www.example.com/blog/hello-world
# expect: 200The first command impersonates GPTBot; the second impersonates a normal browser. If both return 200, your matcher is excluding the path or the regex isn't matching — check the middleware config before assuming the code is wrong.
3. Check X-Robots-Tag on your non-HTML assets:
curl -sI https://www.example.com/downloads/report.pdf | grep -i x-robots-tagYou should see your configured value in the response headers. If it's missing, the source pattern in your next.config.js doesn't match the actual URL path — the most common cause is a trailing-slash or locale-prefix mismatch.
4. Re-test after every deploy. Next.js deployments can change which file serves /robots.txt if someone adds a static file to public/ later. Make this four-command check part of your deploy checklist, and run the URL through the robots.txt tester with a real bot token for a second opinion.
FAQ
Should I use robots.ts or public/robots.txt in Next.js?
Use app/robots.ts if you need dynamic rules (different per environment, generated from a CMS or database). Use public/robots.txt if your rules are static — it's simpler and has no build dependency. Never use both: the static file wins and your robots.ts will be silently ignored.
Can Next.js middleware reliably block AI crawlers?
It can refuse requests whose User-Agent header matches known AI-bot tokens, which stops honest bots that ignore robots.txt and lazy scrapers. It cannot stop scrapers that spoof their user agent, and it can't do proper reverse-DNS verification cheaply. Treat it as enforcement for the abusive remainder, not as your primary control.
How do I block AI bots from my Next.js API routes?
Put Disallow: /api/ in the wildcard (*) group of your robots.txt — that covers every bot without a specific group. For enforcement, add /api/ paths to middleware matcher's scope and return 403 for AI-bot user agents.
Does X-Robots-Tag stop AI training on my content?
No. X-Robots-Tag supports indexing directives (noindex, nosnippet) that keep content out of search indexes and AI Overviews, but there is no widely-honored AI-training directive in the header. Use the Google-Extended, GPTBot and ClaudeBot robots.txt tokens for training opt-outs.
How do I keep my Next.js staging site out of search results?
Branch your robots.ts on the environment: return Disallow: / for everything when not in production (e.g. process.env.VERCEL_ENV !== 'production' on Vercel). Also consider password-protecting staging entirely — robots.txt is a request, not access control.
Why isn't my updated robots.txt showing live?
Three usual suspects: you have both robots.ts and public/robots.txt (static wins); the CDN or edge cache is serving a stale copy (purge it); or the deployment didn't include the change. Fetch /robots.txt in incognito and test it with a robots.txt tester to see what crawlers actually receive.
