Articles about "robots.txt wildcards"

Every robots.txt directive explained in plain English: Allow, Disallow, Crawl-delay, Sitemap, wildcards, and the common mistakes that accidentally block Google. With copy-paste examples you can use today.

Robots.txt wildcards explained: how the asterisk and dollar sign work under RFC 9309, how crawlers pick the winning rule, and the five wildcard mistakes that block the wrong pages.