Short answer: BigCommerce lets you edit robots.txt from Settings > General > Storefront Robots.txt (older accounts show it under Store Setup > Store Settings > Search Engine Robots). Add one block per AI training crawler — GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, and Meta-ExternalAgent — keep Googlebot, Bingbot, OAI-SearchBot, and PerplexityBot allowed, and keep the sitemap line pointing at your store's automatic sitemap at /xmlsitemap.php. Then check yourstore.com/robots.txt to confirm the edit is live, because BigCommerce has moved this setting more than once and old tutorials point at the wrong screen.
BigCommerce is a strong platform for crawler control. You get a real robots.txt editor, an automatically generated sitemap, and SEO settings for products and categories. The weaknesses are the ones shared by every hosted store platform: the setting has moved between interface versions, the default file is easy to break by accident, and store owners often confuse blocking a crawler with removing a page from search results. This guide walks through the current setup, a ready-to-use file, and the store-specific traps.
Finding the robots.txt editor on BigCommerce
BigCommerce has relocated its robots.txt editor over the years, so start by checking the current path. In the current control panel, go to Settings > General > Storefront Robots.txt. On older interfaces the same editor lives under Store Setup > Store Settings, in the Search Engine Robots section. If you follow an old tutorial to a screen that no longer exists, use the current path — the editor is the same, only the navigation changed.
When you open the editor, you will see the store's current robots.txt content, including BigCommerce's defaults. Read the whole file before changing anything. The defaults exist for a reason: they keep crawlers out of cart, checkout, account, and internal search pages that would otherwise waste crawl budget and create duplicate-content noise. Your job is to add AI-crawler rules to that file, not to replace it.
Save after editing, then verify from the outside. Open yourstore.com/robots.txt in a browser and confirm your new rules are present. If they are not, you edited a different storefront or channel — BigCommerce supports multiple storefronts, and the robots.txt setting can differ per channel. Check which channel you edited before assuming the save failed.
What your BigCommerce sitemap does for you
BigCommerce generates an XML sitemap automatically at /xmlsitemap.php, and it updates itself as you add or remove products, categories, and pages. You do not edit this file by hand. What you should do is declare it in robots.txt with a sitemap line, and submit the same URL in Google Search Console and Bing Webmaster Tools so search engines find it directly.
The sitemap matters for AI crawlers too. Search crawlers behind AI answers — OAI-SearchBot for ChatGPT, PerplexityBot for Perplexity — discover pages through the same sitemap and crawl paths as Googlebot. A store with a broken or missing sitemap line is harder to discover for everyone, not just for Google. If your robots.txt edit accidentally drops the sitemap line, put it back before anything else.
Which AI crawlers to block, and which to keep
For most stores, the right split is simple. Block the crawlers that collect content for AI training. Keep the crawlers that bring shoppers through search and AI answers.
Block for training opt-out: GPTBot (OpenAI training), ClaudeBot (Anthropic training), CCBot (Common Crawl's open dataset, used by many AI labs for training), Meta-ExternalAgent (Meta AI training), Google-Extended (a control token that opts content out of Gemini training and grounding, not a crawler), and Applebot-Extended (Apple's training opt-out token).
Keep allowed for visibility: Googlebot (Google Search, which feeds AI Overviews and AI Mode), Bingbot (Bing, which grounds Microsoft Copilot), OAI-SearchBot (ChatGPT search answers), and PerplexityBot (Perplexity answers, which link back to your product pages). Perplexity says its bot is not used to crawl content for foundation models, so allowing it is a visibility decision, not a training decision.
Handle with care: user-triggered fetchers such as ChatGPT-User and Perplexity-User visit a page because a person asked about it. Their operators say robots.txt rules may not apply to those visits. A rule naming them is worth adding as a statement of intent, but it will not reliably stop a fetch a shopper triggered.
A ready-to-use robots.txt setup for BigCommerce
Here is a starting file for a store that wants search visibility and AI citations but not training use. Add these blocks to your existing BigCommerce file, keeping the default store rules and the sitemap line:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / Sitemap: https://yourstore.com/xmlsitemap.php
Notice what this file does not do. It does not name Googlebot, Bingbot, OAI-SearchBot, or PerplexityBot, because unnamed bots fall under the store's existing rules and stay allowed. It does not block product, category, or blog pages — blocking those would remove the pages that actually earn traffic. And it does not touch the default blocks on cart, checkout, account, and search-result pages. Those defaults protect your crawl budget; removing them to “simplify” the file would let every crawler, AI or not, burn requests on pages that can never rank.
If your store sells products where AI training use is genuinely not a concern — for example, a store whose content is entirely manufacturer descriptions already published everywhere — you may choose to allow training crawlers too. That is a business decision, not a technical one. Write it down either way, so a future editor does not flip the policy by accident.
Store-specific traps on BigCommerce
Faceted search and filter URLs. BigCommerce stores generate large numbers of filtered and sorted URLs. The default file already discourages crawling of many of these, but if your store has heavy faceting, check that AI crawlers are not chewing through thousands of near-duplicate filter pages. The training bots you block will stop, but the allowed search crawlers still need sensible filter handling. This is a crawl-budget issue that affects every bot, not just AI ones.
Blocking a product page by accident. The most expensive robots.txt mistake on a store is blocking a product or category path. A single broad rule — for example, disallowing a directory that happens to contain product images or a category slug — can remove revenue pages from search. Before saving, run the file through a tester against your top ten product and category URLs and confirm each one is allowed for Googlebot.
Confusing robots.txt with noindex. A robots.txt block stops crawling; it does not remove a page from search results, and it can actually prevent a noindex tag from being seen. If a page should leave search entirely — a discontinued product page you do not want indexed, a test page — use BigCommerce's page-level controls or a noindex approach, and do not also block the page in robots.txt. Use robots.txt for crawler policy, and use indexing controls for indexing.
Multi-storefront confusion. If you run more than one storefront or channel, each can have its own robots.txt settings. A rule added to one storefront does not protect the others. Audit every storefront separately, and keep a note of which file belongs to which domain.
Theme and app changes. Some BigCommerce apps and theme changes alter SEO settings or inject their own rules. After installing an SEO app or changing themes, re-read your live robots.txt file. Apps that promise “automatic SEO” sometimes rewrite the file without asking.
Testing your BigCommerce robots.txt file
- Fetch the live file. Open yourstore.com/robots.txt and read it. Confirm your AI-crawler blocks are present, the defaults are intact, and the sitemap line is correct.
- Test the bots that matter. Run the file through a robots.txt tester for Googlebot, GPTBot, OAI-SearchBot, and PerplexityBot. You want Googlebot, OAI-SearchBot, and PerplexityBot allowed, and GPTBot disallowed.
- Test your money pages. Check your best-selling product pages and top categories against Googlebot in the tester. All of them should be allowed.
- Check Search Console. A few days after the change, look for coverage changes or crawl errors. A sudden drop in indexed product pages is a red flag worth investigating the same day.
- Recheck after platform changes. BigCommerce moves settings between releases. After any major admin change, theme change, or app install, confirm the file is still what you expect.
What robots.txt cannot do for a store
Robots.txt cannot stop a bot that ignores it, cannot remove content already collected, and cannot protect pages that must stay private. Wholesale pricing pages, dealer portals, and unreleased product pages should live behind login or password protection, not behind a polite request in a text file. For stronger bot enforcement, a firewall in front of the store can block bots by user agent and verified IP ranges — a separate project that needs DNS control and testing, not a robots.txt edit.
Also remember that blocking AI training crawlers is a forward-looking signal. It says your content should not be used for future training. It does not claw back content already in a training dataset, and it does not stop a shopper from pasting your product description into a chatbot themselves. Set expectations accordingly, and do not sell a robots.txt edit to a client as a complete content-protection solution.
FAQ
Where do I edit robots.txt on BigCommerce?
Go to Settings > General > Storefront Robots.txt in the current control panel. Older accounts show the same editor under Store Setup > Store Settings > Search Engine Robots. Save, then verify at yourstore.com/robots.txt.
What is the BigCommerce sitemap URL?
BigCommerce generates it automatically at yourstore.com/xmlsitemap.php. Declare it in robots.txt and submit it in Google Search Console and Bing Webmaster Tools.
Does blocking GPTBot hurt my BigCommerce store's ChatGPT visibility?
No. GPTBot collects content that may be used for training. ChatGPT search answers come from OAI-SearchBot's index. Blocking GPTBot alone leaves your eligibility for ChatGPT search answers intact.
Should I block PerplexityBot on my store?
Usually not. PerplexityBot surfaces and links product pages in Perplexity answers, and Perplexity says it is not used to crawl content for foundation models. Blocking it removes a source of referral traffic without a training benefit.
Can I block AI crawlers on just one BigCommerce storefront?
Yes. Robots.txt settings can differ per storefront and channel. Edit the file for the storefront you mean to change, and audit each storefront separately.
Will robots.txt stop scrapers from copying my product descriptions?
No. Scrapers that ignore robots.txt will keep scraping. Robots.txt is a request that well-behaved crawlers honor. For copied content, use copyright enforcement and, for private pages, access control.
Build your file with the free robots.txt generator and verify it with the robots.txt tester. For comparison, see blocking AI crawlers on Shopify and blocking AI crawlers on WordPress. OpenAI documents its crawler tokens at developers.openai.com/api/docs/bots.
