Site tools
robots.txt
Control exactly how search engine crawlers interact with your site using the built-in robots.txt tools.
Managing crawl rules
A robots.txt file provides instructions to web robots about which pages they are allowed to crawl.
1
Go to Bricks SEO > Settings, then click the Tools tab.
2
Locate the robots.txt editor to add, modify, or remove your crawl directives.
3
Save your changes. The file is served dynamically, so it does not create a physical file in your root folder.
Automatic sitemap injection
Bricks SEO seamlessly integrates your dynamic sitemap with your robots.txt file to ensure perfect crawler discovery.
Your sitemap URL is automatically appended to the end of the file.
This guarantees that Googlebot easily finds your sitemap index.
No manual configuration is required for XML sitemap discovery.
Exercise caution
Be very careful when editing your robots.txt file manually. A single incorrect Disallow rule can accidentally block search engines from indexing your entire site.