Create a valid robots.txt to guide search engine crawlers.
Generated locally in your browser.
Understanding your result
robots.txt is a guideline crawlers usually respect; it is not a security measure. Save it at your site root.
Formula and method
The file lists a User-agent line, Allow/Disallow rules and an optional Sitemap directive — the standard robots.txt format.
Worked example
Disallowing /admin/ produces “User-agent: *” followed by “Disallow: /admin/”.
How to use this tool
- Choose whether to allow all crawlers.
- Add any paths to disallow.
- Add your sitemap URL, then copy the file.
Common mistakes to avoid
- Accidentally disallowing your whole site with “Disallow: /”.
About the Robots.txt Generator
Generate a valid robots.txt file to tell search engine crawlers which paths to avoid and where your sitemap lives.
Explore all SEO and Website Tools tools
Frequently asked questions
Where does robots.txt go?
At the root of your domain, for example https://example.com/robots.txt.