48:T426,

robots.txt

robots.txt tells search engine crawlers which pages they can and cannot access.

In this section

What is robots.txt
Check details
Common issues
Platform guides: WordPress · Shopify · Joomla

Quick facts

Location: https://yoursite.com/robots.txt
Format: Plain text with User-agent, Allow, Disallow, and Sitemap directives
Specification: robotstxt.org

2b:["$","main",null,{"id":"main","tabIndex":-1,"children":[["$","div",null,{"className":"breadcrumb","children":[["$","a",null,{"href":"/","children":[" ","Documentation"]}],["$","span",null,{"children":[" / ","Governance Checks / robots.txt"]}]]}],false,["$","article",null,{"dangerouslySetInnerHTML":{"__html":"$48"}}],["$","nav",null,{"className":"pagination","aria-label":"Adjacent pages","children":[["$","a",null,{"href":"/governance/governance-scoring/","children":["← ","Governance Scoring"]}],["$","a",null,{"href":"/governance/robots-txt/what-is-robots-txt/","children":["What is robots.txt"," →"]}]]}],["$","footer",null,{"children":["Flowpane · ","Customer documentation",["$","a",null,{"href":"https://crawler.flowpane.com","children":"Crawler identity"}],["$","nav",null,{"aria-label":"Legal","children":[["$","a",null,{"href":"/legal/privacy-policy/","children":"Privacy"}],["$","a",null,{"href":"/legal/terms-of-service/","children":"Terms"}],["$","a",null,{"href":"/legal/acceptable-use/","children":"Acceptable use"}],["$","a",null,{"href":"/legal/data-processing/","children":"Data processing"}]]}]]}]]}] 2c:["$","aside",null,{"className":"toc","children":["$","nav",null,{"aria-label":"On this page","children":[["$","h2",null,{"children":"On this page"}],[["$","a","in-this-section",{"className":"depth-2","href":"#in-this-section","children":"In this section"}],["$","a","quick-facts",{"className":"depth-2","href":"#quick-facts","children":"Quick facts"}]]]}]}]