How to Block AI Crawlers on Weebly (2026): Robots.txt Limits, Page Hiding & Meta Tag Fixes

Here is the honest answer up front: you cannot edit the robots.txt file on Weebly directly, but you can still limit what AI crawlers see. You can hide individual pages from search engines, hide your whole site while it is not ready, add noindex meta tags to specific pages, and, if you need true per-bot control, route your domain through Cloudflare and block AI bots there. Each method does something different, and picking the wrong one can hide your site from Google too. This guide walks through all four methods, step by step, so you can block AI training bots without losing the search traffic you actually want.

If you run a Weebly site and you have noticed your content showing up in AI answers, or you simply do not want your work used to train AI models, you are not alone. Many site owners feel the same way. The good news is that Weebly gives you enough controls to cover most cases. The bad news is that none of them is a single switch labeled "block all AI bots." You have to combine a few settings, and you have to understand what each one really does.

Why Weebly Makes This Harder Than WordPress

On a self-hosted WordPress site, you can open the robots.txt file, edit it, and add a rule for each bot you want to block. Weebly does not work that way. Weebly is a hosted platform. It manages the server for you, and it does not give you access to server files. When you publish a Weebly site, the platform creates a robots.txt file for you automatically, and you cannot open that file and rewrite it.

That single limit shapes everything else in this guide. Every method below is a workaround for it. Some methods work through Weebly's own settings. One method works around Weebly entirely by putting Cloudflare in front of your domain. None of them is perfect, and that is worth saying plainly before you start changing settings.

There is also one important fact to keep in mind through the whole process: robots.txt is a request, not a lock. Well-behaved bots from OpenAI, Anthropic, Google, and Meta generally respect it. A bot that ignores robots.txt will not be stopped by any of the Weebly settings below. That is why the Cloudflare method matters for owners who want stronger enforcement.

Before You Block Anything: Training Bots vs. Search Bots

Not every AI bot does the same job, and blocking the wrong ones can hurt you. AI bots fall into two broad groups.

Training bots crawl the web to gather data used to train AI models. GPTBot from OpenAI, ClaudeBot from Anthropic, CCBot from Common Crawl, PerplexityBot, Google-Extended, Meta-ExternalAgent, and Bytespider from ByteDance are the names most site owners mean when they say "AI crawlers." Blocking these bots means your content is less likely to be used as training data.

Search bots, on the other hand, crawl pages so they can be cited or surfaced in AI search answers. OAI-SearchBot from OpenAI and Claude-SearchBot from Anthropic are examples. If you block search bots, your pages are less likely to appear as cited sources in AI search results. For many businesses, that citation traffic is valuable. Blocking a search bot does not just stop training; it can also remove you from the answers your potential customers read.

So before you touch any setting, decide what you actually want. Do you want to stop AI training use of your content? Do you want to disappear from AI search answers too? Or do you want to block training bots while staying visible in AI search? Your answer decides which methods below make sense for you. If you only care about training, you want per-bot blocking, which points you toward the Cloudflare method. If you just want a few pages kept private, the page-hiding method is enough.

Bots to Block If You Want to Stop AI Training

• GPTBot — OpenAI's training crawler. Blocking it signals that you do not want your content used for model training.
• ClaudeBot — Anthropic's training crawler. Same idea: it gathers content for training.
• CCBot — Common Crawl's bot. Its dataset is widely used by AI developers, so blocking it matters if you want broad training protection.
• PerplexityBot — Perplexity's crawler. It is used for both search and training-related purposes, so check its current documentation before deciding.
• Google-Extended — Google's control for AI training use. Blocking it tells Google not to use your content to improve its AI products, while leaving normal Google Search crawling alone.
• Meta-ExternalAgent — Meta's AI training crawler. Blocking it covers Meta's training use.
• Bytespider — ByteDance's crawler, used for AI training data collection.

Bots to Allow If You Want AI Search Citations

• OAI-SearchBot — OpenAI's search crawler. It helps your pages get cited in ChatGPT search answers. Block it only if you want out of AI search entirely.
• Claude-SearchBot — Anthropic's search crawler. It serves a similar citation role for Claude's search features.
• Googlebot — Regular Google Search crawling. Do not block this unless you want to leave Google Search altogether.

The distinction matters because Weebly's own settings are blunt. Hiding a page from search engines hides it from everyone, including Google and AI search bots. That is fine for a private page, but it is a bad way to block only training bots on a public page. Keep that trade-off in mind as you read the methods below.

Method 1: Hide Individual Pages from Search Engines

This is the most useful Weebly setting for most owners. It lets you keep a page live for visitors who have the link, while telling search engines not to index it. It is the right tool for draft pages, member-only content, duplicate pages, or pages you simply do not want in search results or AI answers.

Step by step

1. Sign in to Weebly and open the site you want to change.
2. Click the Pages tab at the top of the editor.
3. Click the page you want to hide.
4. Open that page's SEO Settings.
5. Check the box labeled "Hide this page from search engines."
6. Publish your site so the change goes live.

When you check that box, Weebly adds the page to its robots.txt file. Search engines that respect robots.txt will then skip the page. In practice, this is also how you keep a page out of AI search results, since AI search bots generally follow the same rules for pages excluded from search.

A few practical notes. First, this setting does not password-protect the page. Anyone with the direct link can still open it. If the content must stay truly private, you need Weebly's membership or password features, not this setting. Second, the change only takes effect after you publish. Saving a draft is not enough. Third, if a page was already indexed, it can take time for search engines to drop it after you hide it. Be patient, and check back after a few weeks rather than expecting it to vanish overnight.

Use this method for: a pricing page you are reworking, a thank-you page, an internal resource page, or any page you want visitors to reach only through a direct link or your own navigation.

Method 2: Hide Your Entire Site from Search Engines

Weebly also offers a site-wide version of the same idea. You can find it under Settings, in the SEO section, as an option labeled "Hide site from search engines." Checking it tells search engines to stay away from your whole site.

Step by step

1. In the Weebly editor, click Settings.
2. Open the SEO section.
3. Check the box labeled "Hide site from search engines."
4. Publish your site.

This setting is powerful, and that is exactly why you should use it carefully. It blocks all search engines, including Google. It does not pick out AI bots and leave Google alone. If you turn it on for a live business site by mistake, your search traffic can drop because search engines will stop indexing your pages.

The right time to use it is before launch. If your site is still under construction, or if it is a private project that should never appear in search, this setting is the simplest way to keep it hidden. The wrong time to use it is as a quick fix for AI crawlers on a live site. For a live site, hiding everything is far too broad.

If you turned this on in the past and your site is not showing in Google, check this setting first. It is one of the most common causes of a Weebly site disappearing from search results, and it is also one of the easiest to fix: uncheck the box, publish, and give search engines time to return.

Method 3: Add Robots Meta Tags to Individual Pages

The third Weebly method gives you page-level control through meta tags. A robots meta tag sits in the header code of a page and tells crawlers what they may do with that page. The two values most owners need are noindex and nofollow.

A noindex tag tells search engines not to index the page. A nofollow tag tells them not to follow the links on the page. You can use them together or on their own, depending on what you want. For example, a page you want kept out of search but whose links you still want crawlers to follow would use noindex without nofollow. A page you want fully ignored would use both.

Step by step

1. Open the Pages tab and select the page.
2. Open that page's SEO Settings.
3. Find the header code box, where you can add code to the page header.
4. Add a robots meta tag for the behavior you want. A typical noindex tag is a meta element with name set to robots and content set to noindex. A combined tag uses content set to noindex, nofollow.
5. Publish the site.

Meta tags are useful because they apply to the page itself, not to the site's robots.txt file. That makes them a good fit when you want fine control over one page without changing anything else. They are also the standard way to remove a page from an index after it has already been indexed, since crawlers read the tag when they visit the page.

There are limits, though. A meta tag cannot block one bot and allow another unless you use bot-specific tags, and Weebly's header code box is a per-page tool, so applying different rules across many pages takes manual work. Meta tags also only work if the crawler visits the page and reads the tag. A bot that ignores meta tags will not be stopped by them. Still, for pages you want out of search and AI answers, a noindex tag is a solid, well-supported choice.

If you are testing whether your meta tags are in place, open the live page in your browser and view the page source. Look for the robots meta tag in the header section. If it is missing, you may have added it to the wrong page or forgotten to publish. Both mistakes are easy to make, and both are easy to fix.

Method 4: Block AI Bots Properly with Cloudflare

Methods 1 to 3 are the best Weebly itself offers. But none of them gives you per-bot control. You cannot say "block GPTBot but allow Googlebot" inside Weebly, because you cannot edit robots.txt. If that level of control is what you need, you have to put something in front of Weebly that can make those decisions. Cloudflare is the most practical option for most Weebly owners.

The idea is simple. You point your domain's DNS to Cloudflare, so visitor and bot traffic passes through Cloudflare before it reaches your Weebly site. Cloudflare can then block requests based on the bot's user-agent string, which is the name the bot announces when it visits. You write one rule per bot, or a single rule that matches several bot names, and Cloudflare stops those requests before they ever reach Weebly.

Step by step

1. Sign up for a Cloudflare account and add your domain.
2. Change your domain's nameservers to the ones Cloudflare gives you, following Cloudflare's setup prompts.
3. Make sure your domain's DNS records still point to Weebly, so your site keeps loading normally.
4. Wait for the DNS change to spread. This usually takes minutes to a few hours, but it can take up to a day.
5. In Cloudflare, create a firewall rule that blocks requests whose user-agent contains the names of the AI training bots you want to stop, such as GPTBot, ClaudeBot, CCBot, Google-Extended, Meta-ExternalAgent, PerplexityBot, and Bytespider.
6. Test the rule by checking that your site still loads normally in a browser and that the blocked bot names are being stopped in Cloudflare's firewall event log.
7. Keep the search bots you want, such as OAI-SearchBot and Claude-SearchBot, off the block list if you want to stay visible in AI search answers.

Cloudflare's free plan includes firewall rules, which is why this route is realistic for small sites. You do not need to pay for an advanced bot management product just to block a handful of named training bots. That said, changing nameservers is a bigger step than checking a box in Weebly. If you get the DNS records wrong, your site can go offline, so copy the existing records carefully before you change anything, and do this at a quiet time when you can check that the site still loads.

Cloudflare also has a built-in option to block AI bots, which is worth checking in your Cloudflare dashboard. Bot names and dashboard options change over time, so verify the current wording in Cloudflare's own docs rather than relying on an old tutorial. The firewall rule approach above still works even if the built-in toggle changes, because it blocks on the user-agent names directly.

This method is the only one in this guide that can block training bots while leaving Google and AI search bots alone. If your goal is precise control, this is the method to use. The Weebly settings are still useful alongside it — for example, hiding a private page and blocking training bots at Cloudflare at the same time is a sensible combination.

What Weebly Cannot Do (Honest Limits)

It helps to be clear about what is not possible on Weebly, so you do not waste time looking for a setting that does not exist.

• You cannot edit robots.txt directly. There is no file editor, no server access, and no official way to rewrite the file Weebly generates. Page hiding adds pages to it, but you cannot write custom bot rules in it.
• You cannot block one bot and allow another inside Weebly. The built-in settings are all-or-nothing at the page or site level. Per-bot rules need Cloudflare or another front-end service.
• You cannot force a bad bot to obey. Robots.txt, meta tags, and page hiding are all requests. A crawler that ignores them will keep crawling. Cloudflare blocking is stronger because it stops the request, but even it relies on the bot identifying itself honestly.
• You cannot undo training that already happened. Blocking a bot today does not remove content it already collected. If your content was already used for training, blocking only prevents future collection.
• You cannot rely on a single setting to cover everything. A page hidden from search engines may still be visited directly, and a bot blocked at Cloudflare may still reach your site if it comes through a different route. Layering methods is the realistic approach.

None of this means Weebly is a bad choice. It means Weebly trades control for simplicity. For many small sites, that trade is fine. For owners who need precise bot control, the answer is to add Cloudflare, not to fight Weebly's limits.

A Practical Setup for Most Weebly Owners

If you want a simple plan that covers the common cases, here is the combination most owners should start with.

1. Hide any pages that should never appear in search, using the per-page setting. Publish the change.
2. Add noindex meta tags to pages that were already indexed and need to be dropped from search results.
3. Leave the site-wide hide setting off unless your site is pre-launch or private.
4. If you want to block AI training bots without losing search visibility, set up Cloudflare and block the training bot names listed earlier, while leaving search bots alone.
5. Check your work: confirm hidden pages show the right behavior, confirm your live site still loads, and confirm your main pages are still indexed by Google after a few weeks.

For a deeper look at the same problem on a platform where you can edit robots.txt directly, see our WordPress guide: How to Block AI Crawlers on WordPress. If you want help writing a robots.txt file for a site where you do control the file, try the free robots.txt generator and check the result with the robots.txt tester.

Finally, keep your list of blocked bots up to date. Bot names change, new bots appear, and old ones get renamed. Check OpenAI's bot documentation at platform.openai.com/docs/bots and Anthropic's crawler documentation from time to time, and update your Cloudflare rules when a new training bot shows up in your logs. Blocking AI crawlers is not a one-time job; it is a setting you review every few months, the same way you review any other part of your site's SEO.

Frequently Asked Questions

Can I edit robots.txt on Weebly?

No. Weebly creates a robots.txt file automatically when you publish, and it does not give users a way to edit the file directly. You can influence it by hiding pages, which adds those pages to the file, but you cannot write custom rules for individual bots. For per-bot rules, use Cloudflare in front of your domain.

Does hiding a page from search engines block AI crawlers?

It blocks well-behaved crawlers from indexing that page, including most AI search bots, because hiding a page adds it to robots.txt. It does not block a bot that ignores robots.txt, and it does not stop someone with the direct link from opening the page. For stronger enforcement, combine page hiding with Cloudflare blocking.

Will hiding my site from search engines hurt my Google ranking?

Yes, if you leave it on for a live site. The site-wide hide setting blocks all search engines, including Google. Only use it for pre-launch or private sites. If your live site is missing from Google, check this setting first, turn it off, publish, and allow time for search engines to return.

Should I block AI search bots like OAI-SearchBot and Claude-SearchBot?

Only if you want to disappear from AI search answers. Search bots are what let your pages get cited in AI answers. If you only object to training use, block the training bots such as GPTBot and ClaudeBot, and leave the search bots alone. Blocking everything costs you citation visibility you may actually want.

Is robots.txt enough to stop AI training on my content?

For well-behaved bots, it is the standard signal, and major AI companies say they respect it. But robots.txt is a request, not enforcement. A bot that ignores it will not be stopped. If enforcement matters to you, block the bot user-agents at Cloudflare, where the request is stopped before it reaches your site.