robots.txt
Articles about "robots.txt"
33 articles found
Technical SEO
XML Sitemaps in 2026: The Complete Guide to Faster Google Indexing
A clean XML sitemap gets your pages discovered faster. Learn the format, what to include, how to submit to Google, and the mistakes that slow indexing down.
AI Search
How to Get Cited by Perplexity: A Practical Guide for 2026
Perplexity actually sends traffic through its citations. Learn how to get cited: crawler access, quotable content structure, authority signals, and measurement.
Technical SEO
Technical SEO Checklist: 50 Items Every Site Must Pass in 2026
The full 50-item technical SEO checklist for 2026: crawlability, Core Web Vitals, mobile, structured data, security, and AI-search readiness in priority order.
Content Strategy
How to Protect Your Blog Content from AI Scraping: A Creator's Guide (2026)
Stop AI crawlers training on your blog: robots.txt rules for GPTBot and ClaudeBot, Cloudflare one-click AI blocking, and the search bots you must never block.
New Standards
llms.txt vs robots.txt: What Each File Does (and Why You Probably Need Both)
robots.txt tells crawlers where NOT to go. llms.txt tells AI systems what your site IS. Opposite jobs — here's what each file does and how to use both.
Technical SEO
How to Block Mistral AI Crawlers (MistralAI-User) With robots.txt in 2026
Learn how to block Mistral AI crawlers (MistralAI-User) with robots.txt in 2026 — exact code, the three Mistral tokens, and when blocking helps or hurts.
Technical SEO
How to Block Amazonbot With robots.txt in 2026 (Exact Code + Verification)
Learn how to block Amazonbot with robots.txt in 2026: the exact code to block all three Amazon crawlers, selective allow rules, and how to verify the block worked.
New Standards
How to Add llms.txt to WordPress: 3 Methods That Work in 2026
Add llms.txt to WordPress with a plugin, a manual upload, or a dynamic code snippet. Step-by-step instructions for all three methods, plus the mistakes that make the file useless.
AI Search
Cloudflare Verified Bots Trust Score 2026: Robots.txt Obedience Now Counts Toward Every Bot's Rating
Cloudflare now grades Verified AI bots on robots.txt obedience. Learn how the trust score works, the Perplexity precedent, and what to do with your robots.txt today.
Technical SEO
Sitemap.xml for AI Crawlers: Do GPTBot, PerplexityBot & Others Actually Read It?
Do AI crawlers like GPTBot and PerplexityBot read sitemap.xml? No company confirms it — but a clean, announced sitemap is still your cheapest path to AI discovery.
New Standards
7 llms.txt Mistakes That Make AI Tools Ignore Your File
Most llms.txt files fail for avoidable reasons: sitemap dumps, dead links, no descriptions, and confusion with robots.txt. Here are the seven mistakes that matter and how to fix each one.
Technical SEO
How to Block AI Crawlers on Static Sites: Hugo, Jekyll, Netlify and Vercel
No plugins, no server code — but static sites can still control AI crawlers. Learn where robots.txt goes on Hugo and Jekyll, how Netlify and Vercel edge rules help, and how to protect preview URLs.
AI Search
How to Block YouBot: Stop You.com's Crawler Without Losing Search Traffic
YouBot feeds You.com answers, not Google rankings. Learn the exact robots.txt token, how to verify real YouBot visits by reverse DNS, and what blocking it does — and does not — do.
Analytics & Data
AI Crawlers Are Eating Your Server Budget: How to Measure and Cut Bot Bandwidth Costs
AI crawlers cost bandwidth, CPU, and hosting money — but only your logs know how much. Learn to measure bot traffic by user-agent, sort crawlers into keep, slow-down, and block buckets, and cut the waste.
AI Search
ChatGPT-User Explained: The OpenAI Bot That Fetches Pages for Real Users
ChatGPT-User fetches a page when a person asks ChatGPT to open it, and robots.txt may not stop it. What it does, what blocking changes, and how to control it safely.
Technical SEO
Free Sitemap Generator: Build a Correct XML Sitemap in Minutes (No Signup)
Use a free sitemap generator to build a correct XML sitemap in minutes, no signup needed. Learn the steps and checks that help search engines find pages.
AI Search
How to Get Cited by Meta AI: Allow the Right Crawler and Earn the Citation
Allow meta-webindexer in robots.txt, keep your pages crawlable and indexable, and write specific, well-sourced answers to become eligible for Meta AI citations.
Advanced Techniques
Free .htaccess Generator (No Signup): Block AI Bots, Redirects & Security Rules in Minutes
Use a free htaccess generator to build bot-blocking rules, redirects, HTTPS forcing, and security settings for Apache sites — no signup, no code writing needed.
Technical SEO
How to Block AI Crawlers on Hugo Static Sites
Learn how to block AI crawlers on Hugo: add robots.txt via the static folder, separate training bots from search bots, control sections, and enforce blocks at your host.
Advanced Techniques
How to Protect PDFs and Downloads From AI Training
Protect your PDFs and downloads from AI training with targeted robots.txt rules, split bot stanzas, and gating for private files — without losing Google Search visibility.
Technical SEO
How to Block AI Crawlers on Vercel (2026): robots.txt, Edge Headers, and Middleware
Block GPTBot, ClaudeBot, and other AI crawlers on Vercel with robots.txt, vercel.json edge rules, and middleware — without hurting your Google rankings.
Technical SEO
How to Block AI Crawlers on Nuxt 3 (2026): Robots.txt, Nitro Routes & Server Logs
Block GPTBot, ClaudeBot, and other AI crawlers on Nuxt 3 with robots.txt, Nitro server routes, and server middleware — without hurting your Google rankings.
SEO Strategy
Robots.txt for News Sites and Publishers: Control AI Crawlers in 2026
News sites face a special AI crawling problem: your content is valuable, time-sensitive, and easy to copy. Learn how to control AI crawlers with robots.txt without losing Google News visibility.
Technical SEO
How to Block AI Crawlers on Gatsby Sites (2026): robots.txt, Plugin, and Edge Rules
Block GPTBot, ClaudeBot, and other AI crawlers on your Gatsby site with a static robots.txt, the robots.txt plugin, or hosting-level edge rules, without blocking Googlebot.
Technical SEO
Allow Googlebot but Block AI Crawlers: Split Your robots.txt Correctly (2026)
Googlebot and AI crawlers run on different tokens. Learn how to allow the search bots you need while blocking GPTBot, ClaudeBot, and the other training crawlers.
New Standards
One Robots.txt Rule to Opt Out of AI Training: What Google, OpenAI, Anthropic and Microsoft Just Agreed
Google, OpenAI, Microsoft and Anthropic agreed on 30 Oct 2026 to support one shared robots.txt signal for opting out of AI training. Here is how the __usage_ai-training token works, what it does not cover, and what to do now.
Technical SEO
How to Block AI Crawlers on Substack: The Toggle, Its Limits, and Your Real Options
Substack does not let you edit robots.txt, but it does have an AI training toggle in Publication Details. Here is where it lives, what it does, what it cannot do, and when to move to a platform you control.
Technical SEO
How to Block AI Crawlers on OpenCart: Robots.txt, .htaccess, Cloudflare and Extensions Compared
OpenCart gives you server access, so you can block AI crawlers four ways: robots.txt, .htaccess, Cloudflare and extensions. Here is what OpenCart's flat-file architecture requires, with tested code and the performance cost to watch.
AI Search
AI Crawlers and EU Copyright: How the TDM Opt-Out Works, What Robots.txt Must Do, and the C-806/26 Case to Watch
In the EU, AI training without a rights reservation is lawful by default. That means the opt-out is the right itself. Here is how Article 4 TDM, Article 53 of the AI Act, robots.txt and the C-806/26 referral fit together.
Technical SEO
How to Block DeepSeekBot: The DeepSeek AI Crawler That Won't Identify Itself (2026)
DeepSeekBot collects your pages for AI training, but DeepSeek publishes no crawler docs, no IP list, and no verification method. Here is the robots.txt rule worth adding — and the firewall layer that actually enforces it.
Technical SEO
How to Block AI Crawlers on Carrd and Static HTML Sites in 2026
Carrd has no robots.txt upload, so blocking AI crawlers needs a workaround: meta tags, Cloudflare in front of your domain, or a move to a static host. Here is each option, plus the five-minute robots.txt for plain HTML sites.
AI Search
AI Crawlers and GDPR: What EU Site Owners Must Know in 2026
The EDPB's 2026 web-scraping guidelines tell AI developers to respect robots.txt, ai.txt, and CAPTCHA signals. Here is what the GDPR actually covers, what it does not, and the six practical steps for EU site owners.
Advanced Techniques
How to Block AI Crawlers with Cloudflare Workers in 2026
A Cloudflare Worker blocks AI crawlers at the edge before they reach your server. Setup takes 20 minutes on the free plan: the code, the bot list, costs, testing, and when to use the one-click control instead.