Short answer: HubSpot CMS lets you edit robots.txt directly from Settings > Website > Pages > SEO & Crawlers, and you can override the defaults for each connected domain. Add one rule block per AI training crawler you want to exclude — GPTBot, ClaudeBot, CCBot, Google-Extended, and Applebot-Extended are the usual five — while leaving Googlebot, Bingbot, and the search crawlers behind AI answers (OAI-SearchBot, PerplexityBot) allowed. Then verify the live file at yourdomain.com/robots.txt, because HubSpot's default rules and per-domain overrides are the two places this setup most often goes wrong.
HubSpot is one of the friendlier closed platforms for crawler control. Unlike builders that give you no file access at all, HubSpot hands you the actual robots.txt editor. The trade-off is that the editor sits inside a settings area most marketers never open, its defaults are shared across domains unless you override them, and a free HubSpot system domain behaves very differently from a connected custom domain. This guide covers the exact clicks, the bot list worth using, the HubSpot-specific traps, and how to test the result.
Where HubSpot keeps your robots.txt file
HubSpot generates a robots.txt file automatically for every site it hosts. To edit it, go to Settings > Website > Pages > SEO & Crawlers (in some accounts the path appears under Content > Pages > SEO & Crawlers). You will see the current file content and an editor where you can change it. If your account hosts more than one domain, select the domain first, then click Override default settings before editing — otherwise your changes may not apply to the domain you think you edited.
That per-domain override is the single most common source of confusion. HubSpot's community threads are full of site owners who edited the file, checked the live URL, and saw nothing change, because they edited the default template rather than the domain-specific version. The rule is simple: no override, no effect on that domain. After saving, always open yourdomain.com/robots.txt in a browser and confirm the change is actually live.
One more HubSpot quirk matters before you start. Sites still running on a free HubSpot system domain (the yoursite.hubspotpagebuilder.com style address) are not meant to be public production sites, and their default robots.txt blocks all crawling. That is by design, not a bug. If you want search visibility or AI citations at all, connect your own domain first — no robots.txt edit on a system domain will fix that.
Decide what you actually want before writing rules
Most HubSpot sites want one of three outcomes, and each one needs a different file:
- Search visibility plus training opt-out. The common choice for marketing sites. Allow search crawlers, block the crawlers that collect content for AI model training. You stay in Google, Bing, ChatGPT search, and Perplexity answers, but you signal that your content should not be used to train models.
- Full AI visibility. Allow everything, including training crawlers. Some brands decide that being part of AI training data is acceptable or even useful. If that is your call, you do not need AI-crawler rules at all — but write the decision down, because a future team member may otherwise add blocks nobody approved.
- Maximum privacy. Block AI crawlers and restrict search indexing on sensitive sections. This fits client portals, partner pages, and internal resources that happen to live on a public HubSpot site. Here you combine robots.txt rules with page-level noindex and, for anything truly private, HubSpot's membership or password protection.
Do not write rules before picking one of these. A file that blocks half the bots at random — usually copied from a blog post about a different platform — is worse than no file, because it creates exactly the accidental search-crawler blocks that cost traffic.
The AI crawlers worth naming on a HubSpot site
HubSpot's editor accepts standard robots.txt syntax, so you name bots by their user-agent token. These are the ones that matter for most HubSpot sites:
- GPTBot — OpenAI's training crawler. Blocking it signals your content should not be used to train OpenAI's foundation models. It does not remove you from ChatGPT search answers; that is a different bot.
- OAI-SearchBot — the crawler behind ChatGPT search. Allow it if you want your pages eligible for ChatGPT search answers. Blocking it is the most common accidental self-inflicted wound in AI-crawler setup.
- ChatGPT-User — fetches a page when a person asks ChatGPT about it. OpenAI says robots.txt rules may not apply to these user-triggered fetches, so treat a rule for it as a request, not a guarantee.
- ClaudeBot — Anthropic's training crawler. Block it to opt out of Anthropic training collection.
- CCBot — Common Crawl's crawler, which builds an open dataset many AI labs have trained on. Blocking it keeps your pages out of future Common Crawl snapshots.
- Google-Extended — not a crawler but a control token. Disallowing it opts your content out of Gemini training and grounding in Gemini Apps and Vertex AI, with no effect on Google Search ranking.
- Applebot-Extended — Apple's training opt-out token. Disallowing it keeps you in Apple search features while opting out of Apple foundation-model training.
- PerplexityBot — Perplexity's search crawler. Allow it if you want Perplexity answers to surface and link your pages. Perplexity says it is not used to crawl content for foundation models.
- Meta-ExternalAgent — Meta's AI training crawler. Block it if you do not want Meta's AI systems training on your site content.
You do not need to name every bot on the internet. A short, deliberate list you understand beats a 60-line paste that nobody on your team can explain.
A ready-to-use robots.txt setup for HubSpot
Here is a sensible starting file for a HubSpot marketing site that wants search visibility and AI citations but not training use. Paste it into the SEO & Crawlers editor, keeping your existing sitemap line, then save:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: * Allow: / Sitemap: https://yourdomain.com/sitemap.xml
Three things to note about this file. First, the search crawlers — Googlebot, Bingbot, OAI-SearchBot, PerplexityBot — are not named, so they fall under the wildcard rule and stay allowed. That is deliberate: naming a bot you mean to allow is unnecessary, and every extra line is a chance to mistype a token. Second, the sitemap line should point at your real HubSpot sitemap URL, which you can confirm in the same SEO & Crawlers area. Third, keep any HubSpot default rules that protect system paths unless you have a reason to remove them. HubSpot's file manager and internal paths are not content you need AI crawlers to see, and stripping defaults you do not understand is how sites accidentally expose things like file-manager directories.
HubSpot-specific traps to avoid
The file-manager directory. HubSpot serves uploaded files from its file manager, and community guidance commonly suggests restricting crawler access to that area. The catch is that many sites host public PDFs, images, and resources there that they do want indexed and linked. Before blocking a file-manager path, check whether any public asset lives under it. If the answer is mixed, allow the specific public subdirectories rather than blocking the whole tree.
Blog and landing-page templates. HubSpot blogs and landing pages can carry their own SEO settings, including page-level noindex. Remember the interaction rule: a page blocked in robots.txt may never be crawled, which means a crawler may never see a noindex tag on it. If a page should leave search results entirely, use the noindex route and do not also block it in robots.txt. If it should stay out of AI training data but remain searchable, use robots.txt rules for the training bots and leave the page crawlable.
Staging and preview domains. HubSpot preview links and staging domains should stay blocked. When a site migrates from a staging domain to a production domain, double-check that the production domain's override is in place and that the staging domain is still blocked. Migrations are a classic moment for a site to go live with the wrong domain's file.
Forgetting the sitemap. After editing robots.txt, confirm the sitemap line is still present and correct. Some editors and migrations drop it silently, and a missing sitemap line slows discovery of new pages for every crawler, not just AI ones.
Testing your HubSpot robots.txt file
Testing takes five minutes and prevents most failures:
- Fetch the live file. Open yourdomain.com/robots.txt and confirm your new rules are actually there, on the right domain, with no leftover default blocking everything.
- Run it through a tester. Paste the file into a robots.txt tester and check it against Googlebot, GPTBot, and OAI-SearchBot. You want Googlebot and OAI-SearchBot allowed and GPTBot disallowed. If any of those three is wrong, fix it before moving on.
- Check Google Search Console. After a day or two, look for crawl errors or sudden drops in indexed pages. A misconfigured file usually shows up fast as a coverage change.
- Watch your own pages. Ask ChatGPT or Perplexity about a topic your site covers and see whether your pages still appear as sources. This is a rough check, not proof, but a sudden disappearance after a robots.txt change is a strong signal.
- Recheck after every HubSpot change. Domain changes, template changes, and plan changes can all reset or alter the file. Put a robots.txt check on your launch checklist for any site change.
If you manage HubSpot for clients, document the file you chose and why. Six months from now, someone will ask why a particular bot is blocked, and “a blog post said so” is not an answer a client accepts.
What robots.txt cannot do on HubSpot
Robots.txt is a request, not a lock. Bots that ignore it will still fetch your pages, and user-triggered fetchers such as ChatGPT-User and Perplexity-User may not honor it at all, because a person asked for the page. If a page must never reach an AI system, do not publish it on a public URL — use HubSpot's membership access, password protection, or a private host. Similarly, robots.txt does not remove content already collected, and it does not delete pages from search indexes. For removal, use noindex and let crawlers visit the page to see it.
For sites that need stronger enforcement, a firewall in front of the site — possible when you control DNS — can block named bots by user agent and verified IP ranges. That is a bigger project than editing a HubSpot setting, and it belongs in a separate plan, not bolted onto a robots.txt edit.
FAQ
Where exactly do I edit robots.txt in HubSpot?
Go to Settings > Website > Pages > SEO & Crawlers (some accounts show it under Content > Pages > SEO & Crawlers). Select the domain, click Override default settings if the option appears, edit the file, and save. Then verify at yourdomain.com/robots.txt.
Why didn't my HubSpot robots.txt change show up?
The usual cause is editing the default template instead of the domain-specific override. Select the domain first and confirm the override is active. The other common cause is checking a system domain, which stays blocked by design — connect a custom domain for a production site.
Does blocking GPTBot remove my site from ChatGPT answers?
No. GPTBot collects content that may be used for training. ChatGPT search answers come from OAI-SearchBot's index. Blocking GPTBot alone leaves your ChatGPT search eligibility intact, which is why the two tokens must be handled separately.
Should I block Google-Extended on HubSpot?
Disallowing Google-Extended opts your content out of Gemini training and grounding, and it does not affect your Google Search ranking or inclusion. If training opt-out is your goal, include it; if you want maximum AI visibility, leave it out.
Can HubSpot block AI crawlers that ignore robots.txt?
Not through robots.txt alone. A request-based file cannot stop a bot that ignores requests. For hard enforcement you need a firewall rule, and for genuinely private content you need access control such as membership or password protection.
Do I need to repeat this for every HubSpot domain?
Yes. Robots.txt is per domain, and HubSpot's override is per domain too. If you run several domains on one account, check each domain's file separately.
Build your file with the free robots.txt generator and verify it with the robots.txt tester. For comparison, see blocking AI crawlers on WordPress and blocking AI crawlers on Squarespace. OpenAI documents its crawler tokens at developers.openai.com/api/docs/bots.
