Audit a website's robots.txt for SEO. Fetches and parses robots.txt to find a redirected or HTML-served file, a 5xx that Google reads as "block everything", a sitewide Disallow: / that blocks the whole site, syntax errors and directives before any User-agent, CSS/JS resources blocked from rendering, missing or unreachable Sitemap: directives, a missing default User-agent: * group, unsupported directives (noindex, nofollow, crawl-delay) that Google ignores, files over the 500 KiB limit, BOM/encoding issues, and duplicate or empty rules. Use when the user asks to audit, analyze, check, validate, debug, or fix a robots.txt file, crawl directives, or what search engines are a…
