Robots.txt rules
Generated robots.txt
Your rules will appear here.How to Use the Robots.txt Generator
Create simple allow and disallow directives for a single user-agent.
1
Set the user-agent
Enter the crawler token to target, or keep * to address all crawlers.
2
Add paths and sitemap
Enter one allowed or disallowed path per line and optionally provide an absolute sitemap URL.
3
Generate and review
Select Generate robots.txt, copy the text, and test the final file before placing it at your site root.
Frequently Asked Questions
Does this upload robots.txt?
No. It only generates text in your browser.
Can robots.txt protect private content?
No. Use authentication or other access controls for private content.