How to Block AI Crawlers on Substack: The Toggle, Its Limits, and Your Real Options
Technical SEO

How to Block AI Crawlers on Substack: The Toggle, Its Limits, and Your Real Options

 Rahul Sharma  Nov 07, 2026  10 min  ~2,129 words

Yes, you can stop AI models from training on your Substack writing, but you cannot do it the way a WordPress or self-hosted site owner would. Substack does not let you edit robots.txt, upload a server file, or add code to your pages. Instead, Substack gives you a single toggle in your publication settings that tells AI companies not to use your content for training. Turn it on, and the platform handles the rest. The catch is that the toggle is blunt, it may also reduce how your work surfaces in AI answers, and it does nothing about AI crawlers that ignore platform settings. This guide explains exactly where the setting lives, what it does and does not do, and what to do if you need stronger control.

The short answer: go to your Dashboard, open Settings, then Publication Details, and switch on the setting that prevents AI models from using your content for training. That is the whole process. There is no robots.txt file to edit, because on Substack you do not own the server. Everything else in this article is about understanding the limits of that toggle and the workarounds that exist around it.

Why Substack is different from a normal website

On a self-hosted site, you control the server. You can place a robots.txt file at the root, add a llms.txt file, set headers, and block individual crawlers by user-agent. On Substack, you rent space on someone else's platform. Your publication lives at yourname.substack.com, or on a custom domain that still points at Substack's servers. You cannot touch the server configuration, and the documented SEO controls in post settings are limited to the title, description, post URL and social preview image. There is no general HTML head editor, no redirect manager, and no robots.txt editor. Any guide that tells you to paste technical markup into your article body to block crawlers is giving you WordPress advice for a platform where it does not apply.

That is not a flaw in Substack. It is the trade-off of a hosted platform: you get publishing, email delivery, payments and hosting handled for you, and in exchange you give up server-level control. The AI-training question is one place where that trade-off becomes visible. The platform has to solve it for you, because you cannot solve it yourself.

The AI training toggle: where it is and what it does

Substack added a publication-level setting for this. In your Dashboard, open Settings, then Publication Details. There you will find two toggles that sit side by side. The first boosts visibility on Google and other search engines. The second prevents AI models from using your content for training. Switch the second one on. Writers have documented both toggles in detail, and a practical Substack SEO checklist confirms the platform's documented controls are limited to the post-level options described below.

What happens when you switch it on is that Substack signals to AI crawlers that your publication's content should not be used for training. The platform controls the robots.txt file for your publication's domain, so it can express the opt-out on your behalf in the machine-readable way the crawlers expect. You do not see the file, and you cannot edit it, but the signal goes out from the same domain your content lives on, which is what matters.

One warning from writers who have used the setting: turning it on may also reduce how your content surfaces in AI tools. The reason is the same one that applies everywhere in crawler control. Training crawlers and answer crawlers are often run by the same companies, and a broad training opt-out can be read by some systems as a general do-not-use signal. If being cited in ChatGPT, Perplexity or Gemini answers is a meaningful source of readers for you, weigh that before switching the toggle on. Our guide to whether you should block AI crawlers walks through that trade-off in detail.

What the toggle does not do

The toggle is a request, not a lock. That is true of every robots.txt-based control, but it is worth stating plainly here because Substack writers cannot add a second layer of enforcement themselves.

  • It does not block search engines. Googlebot and Bingbot are separate from AI training crawlers. Your posts can still be indexed by Google and appear in search results. The first toggle in Publication Details is the one that affects search visibility, and it works independently.
  • It does not remove content already collected. If a crawler already fetched your posts before you switched the toggle on, the opt-out does not reach back in time and delete that copy. It applies to future crawling.
  • It does not stop crawlers that ignore robots.txt. Reputable AI companies honour the signal. Scrapers that ignore robots.txt entirely will ignore this too. You have no server access to add the firewall or user-agent blocking a self-hosted site would use as a second layer.
  • It does not cover AI search answers. As with the new shared training opt-out token the big AI companies agreed in October 2026, a training opt-out is a different decision from a search opt-out. If you want to disappear from AI answers as well, you would need to block the search and retrieval crawlers too, which Substack does not currently let you do per-bot.

Paywalls and private posts: the other lever you actually have

Substack gives you one control that is stronger than any robots.txt rule: the paywall. A post behind a paywall is not freely accessible, and crawlers that cannot read a page cannot train on it. That is not a universal ban on indexing, though. Google's own documentation notes that paywalled content can still be indexed in limited ways, and a paywall changes your relationship with readers far more than it changes your relationship with crawlers. Use the paywall because you want paying subscribers, not as a crawler-blocking tool.

Private and unlisted posts are a cleaner option for content you do not want public at all. A post that is never published publicly is never crawled. If a piece is sensitive enough that you do not want any AI system to see it, do not publish it to the public web in the first place. Email-only delivery to subscribers is the closest Substack gets to a private channel, and it is worth remembering that the email version of your newsletter and the web version are separate things. Sending an email is not the same as publishing a web page, and the web page is what crawlers see.

Custom domains do not change anything

Connecting your own domain to Substack changes your publication's address, not your access to its server configuration. Your robots.txt is still Substack's file, served from Substack's servers, and you still cannot edit it. A custom domain is worth doing for branding and for owning your address, but it does not buy you crawler control. If you are considering a custom domain specifically to gain control over AI crawlers, save your money: it will not do that.

Checking whether it worked

Because you cannot see the robots.txt file, you cannot verify the toggle the way a self-hosted owner would. There are three practical checks you can do instead.

1. Check your publication's robots.txt in a browser

Visit your publication's address with /robots.txt added, for example yourname.substack.com/robots.txt. You will see Substack's platform-level file. You cannot edit it, but you can confirm it exists and see what it says. If the AI-training toggle is working as expected, the file should reflect an opt-out for your publication. If it does not, contact Substack support: platform-controlled problems are theirs to fix, and they are the only ones who can fix them.

2. Watch Google Search Console

Substack lets you verify your publication in Google Search Console from Settings, then Analytics, then Google Site Verification. That gives you data on how Google crawls and indexes your publication. It will not show AI crawler activity directly, but a sudden change in crawl behaviour after you change a setting is a useful signal. Note that some smaller publications report not seeing the verification field until the publication grows, so if the option is missing, keep publishing and check again later.

3. Search for your own work in AI answers

Every few weeks, ask ChatGPT, Perplexity or Gemini a question your posts answer, and see whether your publication is cited. This is slow and imprecise, but it is the only direct test of whether answer engines are still using your content. Our guide to tracking brand mentions in AI answers explains how to do this systematically.

When Substack is not enough: moving to a platform you control

Some writers reach a point where the toggle is not enough. They want per-bot control, a llms.txt file, server-level blocking for crawlers that ignore robots.txt, and the ability to verify all of it in their own logs. Substack cannot give you any of that, and no setting will change it. The honest answer for those writers is to move to a self-hosted platform.

WordPress is the usual destination, and our guide to blocking AI crawlers on WordPress covers the plugin, robots.txt and Cloudflare routes in detail. Ghost is another option, and we have a Ghost guide too. Before you move, weigh the cost honestly: you gain full crawler control, but you take on hosting, security, email delivery and maintenance that Substack currently handles for you. For most writers, the toggle plus a paywall for the most sensitive work is the right balance. Move only if crawler control is genuinely a deciding factor for your publication, not because a guide told you to.

A simple decision checklist

Use this to decide what to do today:

  1. You write publicly and do not mind AI training: leave the toggle off. Your work stays discoverable in search and in AI answers.
  2. You do not want your work training AI models: switch the AI training toggle on in Settings, then Publication Details. Leave the search visibility toggle on.
  3. You want maximum protection for a few pieces: put those pieces behind the paywall, or keep them email-only and unpublished on the web.
  4. You need per-bot control, logs and server blocking: Substack cannot do that. Plan a move to WordPress or Ghost, and use the toggle in the meantime.

If you do move to a platform you control, build your robots.txt with our free robots.txt generator and check it with our robots.txt tester before it goes live. And if you stay on Substack, remember the one thing that matters most: the toggle is the control you have, so use it deliberately, and revisit the decision every few months as the platforms and the standards change.

Frequently asked questions

Can I edit robots.txt on Substack?

No. Substack controls the robots.txt file for your publication's domain, and there is no editor for it. The documented SEO controls in post settings cover the title, description, post URL and social preview image only. The AI training toggle in Settings, then Publication Details, is the platform's way of expressing the opt-out on your behalf.

Does turning on the AI training setting remove my posts from Google?

No. The AI training setting and search visibility are separate toggles. Your posts can still be indexed by Google and appear in search results while the AI training opt-out is on. The two settings work independently.

Will the toggle stop Perplexity or ChatGPT from citing me?

Not reliably. The toggle targets AI training, not AI search answers. Citations in ChatGPT or Perplexity come from search and retrieval crawlers, which are controlled separately. The toggle may reduce how your content surfaces in AI tools, but it is not a guaranteed way to disappear from AI answers.

Does a paywall stop AI crawlers?

Mostly, in practice. A crawler that cannot read a paywalled page cannot train on its content. But a paywall is not a universal ban on indexing, and it changes your reader relationship far more than your crawler relationship. Use it for subscriber revenue, not as your only crawler defence.

Does a custom domain give me control over robots.txt?

No. A custom domain changes your publication's address, not your access to Substack's server configuration. Your robots.txt is still served and controlled by Substack.

What if I want full control over AI crawlers?

Move to a self-hosted platform such as WordPress or Ghost, where you own the server and the robots.txt file. You gain per-bot blocking, llms.txt, server-level enforcement and log access, and you take on hosting and maintenance in return. Until then, use the AI training toggle and the paywall for your most sensitive work.

Related Topics & Keywords

Share this article

Related Articles