Common robots.txt issues

Blanket disallow

An effective Disallow: / asks the matched crawler not to fetch the Site. It may be intentional and is not access control.

Confirm publisher intent before changing it. Flowpane does not automatically reverse a blanket restriction.

Missing sitemap directive

Without a Sitemap: line, search engines must discover your sitemap on their own.

Add a Sitemap: directive only after confirming the exact public sitemap URL. Flowpane will not guess a conventional path as an automatic fix.

Mixed protocols

Mixed origin/protocol declarations can disagree with the effective Site and deserve review.

Preserve an explicit HTTP or custom-port deployment when it is authoritative; otherwise update the declaration to the verified effective URL.

File not found

A 404 is a genuine missing result, distinct from a successful empty file, blocked response or acquisition error.

Flowpane may offer a conservative User-agent: * / empty Disallow: starter. It adds a Sitemap only from same-Site current evidence and does not invent AI policy, Crawl-delay or platform paths.