Free Robots.txt Checker
Enter any website URL and get a plain-English robots.txt score — 12 automated checks for a fetchable robots.txt, real text vs a soft-404 page, user-agent groups, the catch-all rule, a sitemap that resolves, no accidental full-site block, an allowed homepage, AI-crawler rules, a sane crawl-delay, clean syntax, and no placeholder content that decide whether crawlers actually follow your rules. Heuristics are stated as heuristics throughout.
Privacy: we fetch the robots.txt file you point us at plus at most one sitemap URL to run the checks, but we never store any of them. The only thing recorded is an anonymous "grader run" for our own stats. Public websites only.
What the checker checks
Can crawlers read the rules?
A robots.txt that loads, real text instead of a soft-404 page, user-agent groups, the catch-all rule, a sitemap directive that resolves, and no placeholder content.
Is anything blocked by accident?
No accidental full-site block, an allowed homepage path, a sane crawl-delay, clean syntax, and AI-crawler rules for the AI-search channel.
Can crawlers find everything?
The sitemap directive points crawlers at the full URL list, the catch-all group covers the long tail of bots, and AI crawlers get explicit rules.