Running a crawl

Crawls run from the Deep Scan tab of the Sitemap Workbench. Deep Scan is available on Starter and above, to Editors and Admins.

1. Open the Site, go to Sitemaps and select the Deep Scan tab.
2. Click Start crawl.
3. Progress updates every two seconds: pages crawled, current depth and broken links found so far.
4. Click Cancel to stop early. Pages crawled up to that point are kept and analysed.

Crawler behaviour

• Flowpane identifies itself as Flowpane/1.0 (+https://crawler.flowpane.com).
• It respects robots.txt for that user agent and honours Crawl-delay, capped at 10 seconds.
• Crawls start from the homepage and follow internal links within page, depth and execution limits. A bounded crawl may stop before exploring every reachable page.

Rate limits

• One crawl per Site per hour.
• A small number of crawls run across Flowpane at once. If capacity is full, check the status shown in Flowpane; no start time is guaranteed.

See Crawl settings and limits for the per-plan limits.