crawl-delay
Articles about "crawl-delay"
4 articles found
AI Search
How to Block ClaudeBot: Stop Anthropic's AI Crawler With Robots.txt
Block ClaudeBot with two lines of robots.txt. Which of Anthropic's three crawlers to stop, copy-paste rules, verification steps, and trade-offs explained.
Advanced Techniques
Crawl-Delay Explained: Which Crawlers Honor It and Which Ignore It (2026)
Crawl-delay asks crawlers to slow down — but Googlebot ignores it, Yandex dropped it in 2018, and only Bing and a few others obey. Here's the verified support map.
AI Search
ClaudeBot vs Claude-SearchBot vs Claude-User: Which Anthropic Bot Should You Block?
Anthropic runs three crawlers — ClaudeBot, Claude-SearchBot and Claude-User — and blocking the wrong one erases you from Claude's answers. What each does, what blocking costs, and the exact robots.txt setup for no-training-yes-citations.
Analytics & Data
AI Crawlers Are Eating Your Server Budget: How to Measure and Cut Bot Bandwidth Costs
AI crawlers cost bandwidth, CPU, and hosting money — but only your logs know how much. Learn to measure bot traffic by user-agent, sort crawlers into keep, slow-down, and block buckets, and cut the waste.