robots.txt check details

The current evaluator preserves source text while checking parser conformance, effective crawler groups, Allow/Disallow matching, Sitemap directives, AI-crawler policy observations, size and managed-platform evidence.

Findings use Issue, Warning, Recommendation and Information. Warnings and Information are score-neutral. Issues and Recommendations deduct only through explicit score treatment; repeated score families use a bounded multiplier and optional recommendations share an artefact-level cap.

A successful empty file remains present but scores 0. A genuine 404 is a distinct missing result with a conservative starter. Redirects, blocked/WAF responses, destination rejection and other acquisition errors retain their own evidence rather than all becoming “Not Found”. Flowpane does not guess Sitemap URLs or infer CMS/plugin management from generic platform labels.