Robots.txt Generator
Crawl rules, sitemap directives and per-agent overrides. Generated locally in your browser.
Output — robots.txt
About this tool
Create a production-ready robots.txt file. Add crawl rules per user agent, point search engines at your sitemap, and copy the result straight into your project root. Everything runs in your browser - nothing is uploaded.
Frequently asked questions
Where does the generated robots.txt file go?
In the document root of your domain, so it resolves at https://example.com/robots.txt. Crawlers only read the file at that exact location - one placed in a subdirectory is ignored.
Does Disallow remove a page from Google?
No. Disallow stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it. Use a noindex meta tag or an X-Robots-Tag header to keep a page out of the index.
Can I have more than one robots.txt file?
One per host, including the protocol and port. A subdomain such as blog.example.com needs its own file.