Articles about "RFC 9309"

Learn to test your robots.txt online free before going live: validator checks, Search Console verification, server logs, and 7 mistakes testers catch.

Every robots.txt directive explained in plain English: Allow, Disallow, Crawl-delay, Sitemap, wildcards, and the common mistakes that accidentally block Google. With copy-paste examples you can use today.

Robots.txt wildcards explained: how the asterisk and dollar sign work under RFC 9309, how crawlers pick the winning rule, and the five wildcard mistakes that block the wrong pages.