Robots.txt Generator
Configure your robots.txt rules below and download the file.
What Is Robots.txt?
Robots.txt tells search engine crawlers which pages they can and cannot access. It is placed at the root of your website. Always include your sitemap URL so crawlers can find all your pages.