Short answer: Magento 2 — now sold as Adobe Commerce — lets you edit robots.txt from the admin panel at Content > Design > Configuration, by editing the Global or store-view configuration and expanding the Search Engine Robots section. Add one block per AI training crawler — GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, and Meta-ExternalAgent — keep Googlebot, Bingbot, OAI-SearchBot, and PerplexityBot allowed, save the configuration, and flush the cache. Then check yourstore.com/robots.txt, because Magento can serve robots.txt from the admin panel or from a static file, and editing the wrong one has no effect at all.
Magento gives store owners more crawler control than most hosted platforms, because it is self-hosted and the robots.txt editor is built into the admin. That power comes with two traps that hosted platforms do not have: the file can be managed in two different places, and multi-store setups give each store view its own file. This guide covers both, plus a ready-to-use AI-crawler file and the store-specific mistakes that cost product pages their rankings.
Editing robots.txt in the Magento admin
Log in to the Magento admin and go to Content > Design > Configuration. You will see a grid of websites and store views. To change the file for the whole installation, click Edit on the Global row. To change it for one website or store view — the right choice for multi-store setups where domains differ — click Edit on that store view's row instead.
Expand the Search Engine Robots section. You will find three things: a Default Robots dropdown, an Edit custom instruction of robots.txt file text box, and a Reset to Default button. The dropdown controls the meta robots tag applied site-wide — INDEX, FOLLOW is the correct default for a live store, and NOINDEX, FOLLOW or NOINDEX, NOFOLLOW belong only on staging or private environments. The text box is where your robots.txt rules go. Add your AI-crawler blocks there, click Save Configuration, and then flush the Magento cache so the change takes effect.
That last step is easy to forget and hard to diagnose. Magento caches configuration aggressively, and a saved robots.txt change that has not been flushed may not appear on the live URL. If your edit does not show up at yourstore.com/robots.txt, flush the cache before assuming anything else is wrong.
The two places Magento can serve robots.txt from
Before editing, figure out which method your store actually uses, because editing the wrong one changes nothing.
- Admin-managed (most common). The robots.txt content comes from the Search Engine Robots configuration described above. If your store uses this method, the admin edit is the correct edit.
- Static file. If a physical robots.txt file exists in the store's pub directory, the web server may serve that file instead of the admin-managed content. In that case, edit the static file directly, or remove it and let the admin configuration take over. A static file left over from a migration or a developer's manual edit is a classic reason for “my robots.txt edit did nothing.”
Check the live file after every edit. Fetch yourstore.com/robots.txt and read it. If it does not contain your changes, you edited the wrong source, the cache has not been flushed, or a deployment overwrote the file. Do not move on until the live file matches your intent.
Magento's default rules, and what to keep
Magento's default robots.txt starts with a wildcard rule that blocks internal paths most stores never want crawled: the index.php entry point, URLs with query parameters, checkout, app directories, and similar system paths. Those defaults protect crawl budget and keep duplicate-content noise out of search indexes. Keep them.
What you are adding is a set of AI-crawler rules on top of those defaults. Do not replace the default file with a short AI-only file — that would open the system paths to every crawler at once. And watch for one specific danger: if a staging migration, a deployment script, or a hosting change has added a bare “Disallow: /” line anywhere in the file, every bot is blocked, AI or not. Open the live file and search for that line right now if you have never checked. It is the single most expensive robots.txt mistake on Magento, and it produces no error anywhere.
Which AI crawlers to block, and which to keep
For most Magento stores, the split is the same as for other platforms. Block the crawlers that collect content for AI training. Keep the crawlers that bring shoppers through search and AI answers.
Block for training opt-out: GPTBot (OpenAI training), ClaudeBot (Anthropic training), CCBot (Common Crawl's open dataset, used by many AI labs for training), Meta-ExternalAgent (Meta AI training), Google-Extended (the control token that opts content out of Gemini training and grounding, with no effect on Google Search), and Applebot-Extended (Apple's training opt-out token).
Keep allowed for visibility: Googlebot (Google Search, which feeds AI Overviews and AI Mode), Bingbot (Bing, which grounds Microsoft Copilot), OAI-SearchBot (ChatGPT search answers), and PerplexityBot (Perplexity answers, which link back to product pages).
Handle with care: user-triggered fetchers such as ChatGPT-User and Perplexity-User visit a page because a person asked about it, and their operators say robots.txt rules may not apply to those visits. Add a rule for them as a statement of intent if you wish, but do not treat it as a guarantee.
A ready-to-use robots.txt setup for Magento
Add these blocks to the custom-instruction box in the Search Engine Robots section, keeping Magento's default rules intact:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: /
Notice what this file does not do. It does not name Googlebot, Bingbot, OAI-SearchBot, or PerplexityBot, so they fall under the existing rules and stay allowed. It does not block product, category, or content pages. And it does not change the Default Robots dropdown — that dropdown controls the meta robots tag, a separate mechanism from robots.txt, and flipping it to NOINDEX by accident would hide the whole store from search results. Keep the dropdown at INDEX, FOLLOW for a live store.
Magento also offers automatic sitemap submission to robots.txt: under Stores, then Configuration, then Catalog, then XML Sitemap, the Search Engine Submission Settings section has an “Enable Submission to Robots.txt” option. Turning it on appends your sitemap URL to the file automatically. If you use a custom sitemap setup instead, add the sitemap line by hand in the custom-instruction box. Either way, confirm the sitemap line is present in the live file.
Multi-store traps
Magento's multi-store capability is powerful and easy to get wrong for robots.txt. Each store view can have its own robots.txt content, and each store view usually maps to its own domain. A rule added to the Global configuration may be overridden by a store-view configuration, and a rule added to one store view does not protect the others.
Audit every store view separately. Open each domain's robots.txt file and read it. If you run three stores, you should be able to show three files, each correct for its domain. Keep a short note for each store: which method serves its file (admin or static), which AI policy it follows, and when it was last checked. Without that note, a future migration will silently break one store's file and nobody will notice until traffic drops.
Also check staging store views. A staging store that shares a production domain prefix, or a staging domain that search engines can reach, should stay fully blocked. When a staging store becomes production, its robots.txt policy must change with it — a launch checklist item that is easy to skip in the rush of go-live.
Store-specific mistakes to avoid
Blocking product and category pages. A broad rule that disallows a directory containing product images, a category slug, or a layered-navigation path can remove revenue pages from search. Before saving, test your top twenty product and category URLs against Googlebot in a robots.txt tester and confirm all of them are allowed.
Confusing robots.txt with noindex. A robots.txt block stops crawling but does not remove a page from search results, and it can prevent a noindex tag from ever being seen. For a page that should leave search entirely — a discontinued product, a test page — use Magento's page-level controls or a noindex approach, and do not also block the page in robots.txt.
Layered navigation and filter URLs. Magento stores generate huge numbers of filtered URLs. The default file already blocks many query-parameter URLs, but if your catalog is large, check that allowed search crawlers are not burning crawl budget on thousands of filter combinations. This affects every bot, not just AI crawlers, and it is worth a separate crawl-budget review.
Forgetting the cache and the deployment. Magento configuration changes need a cache flush, and file changes made on a server can be overwritten by the next deployment. If your robots.txt content lives in a static file, make sure your deployment process preserves it — or better, move the file management into the admin panel where deployments cannot silently replace it.
Testing your Magento robots.txt file
- Fetch the live file. Open yourstore.com/robots.txt on every store domain and confirm your AI-crawler blocks are present, the defaults are intact, and the sitemap line is correct.
- Test the bots that matter. Run the file through a robots.txt tester for Googlebot, GPTBot, OAI-SearchBot, and PerplexityBot. You want Googlebot, OAI-SearchBot, and PerplexityBot allowed, and GPTBot disallowed.
- Test your money pages. Check best-selling products, top categories, and key landing pages against Googlebot. All of them should be allowed.
- Check Search Console. A few days after the change, look for coverage changes or crawl errors. A sudden drop in indexed product pages deserves same-day investigation.
- Recheck after deployments and migrations. Any deployment, theme change, or store-view change can alter the file. Put a robots.txt check on your release checklist.
What robots.txt cannot do for a Magento store
Robots.txt cannot stop a bot that ignores it, cannot remove content already collected, and cannot protect pages that must stay private. Dealer pricing, wholesale portals, and unreleased product pages belong behind customer-group access or login protection, not behind a polite request in a text file. For stronger bot enforcement, a firewall in front of the store can block bots by user agent and verified IP ranges — a separate project that needs testing, not a robots.txt edit.
Set expectations honestly. Blocking training crawlers is a forward-looking signal: it says your content should not be used for future training. It does not claw back content already in a training dataset, and it does not stop a shopper from pasting your product description into a chatbot. Treat robots.txt as one layer of a content policy, not as the whole policy.
FAQ
Where do I edit robots.txt in Magento 2?
Go to Content > Design > Configuration, click Edit on the Global row or the relevant store view, and expand the Search Engine Robots section. Add your rules in the custom-instruction box, save the configuration, and flush the cache.
Why didn't my Magento robots.txt change appear?
Three usual causes: the cache was not flushed, a static file in the pub directory is being served instead of the admin-managed content, or you edited a different store view than the domain you checked. Check all three before re-editing.
What should the Default Robots dropdown be set to?
INDEX, FOLLOW for a live store. That dropdown controls the meta robots tag, not robots.txt. NOINDEX options belong only on staging or private environments.
Does blocking GPTBot hurt my store's ChatGPT visibility?
No. GPTBot collects content that may be used for training. ChatGPT search answers come from OAI-SearchBot's index. Blocking GPTBot alone leaves your eligibility for ChatGPT search answers intact.
Do I need different robots.txt files for each Magento store view?
Yes, when store views map to different domains or need different policies. Audit each store view's file separately, and keep a note of which method serves each one.
Can robots.txt stop scrapers from copying my catalog?
No. Scrapers that ignore robots.txt will keep scraping. Robots.txt is a request that well-behaved crawlers honor. For copied content, use copyright enforcement, and for private pages, use access control.
Build your file with the free robots.txt generator and verify it with the robots.txt tester. For comparison, see blocking AI crawlers on Shopify and blocking AI crawlers on WordPress. OpenAI documents its crawler tokens at developers.openai.com/api/docs/bots.
