Create a valid robots.txt to guide search engine crawlers.
Generated locally in your browser.
How do you create a robots.txt file?
A robots.txt lists a User-agent line, Allow/Disallow rules and an optional Sitemap directive in the standard format. It is a guideline crawlers usually respect, not a security measure, and should be saved at your site root. For example, disallowing /admin/ produces "User-agent: *" followed by "Disallow: /admin/".
Understanding your result
robots.txt is a guideline crawlers usually respect; it is not a security measure. Save it at your site root.
Formula and method
The file lists a User-agent line, Allow/Disallow rules and an optional Sitemap directive — the standard robots.txt format.
Assumptions and limitations
robots.txt is a guideline that well-behaved crawlers usually respect; it is not a security control and cannot hide or protect private pages, which still need proper access restrictions. Disallowing a path only requests that crawlers skip it. The tool builds the file but does not deploy it. Generation happens in your browser.
Worked example
Disallowing /admin/ produces “User-agent: *” followed by “Disallow: /admin/”.
How to use this tool
- Choose whether to allow all crawlers.
- Add any paths to disallow.
- Add your sitemap URL, then copy the file.
Common mistakes to avoid
- Accidentally disallowing your whole site with “Disallow: /”.
About the Robots.txt Generator
Generate a valid robots.txt file to tell search engine crawlers which paths to avoid and where your sitemap lives.
Who should use this tool
Site owners and developers who need a correctly formatted robots.txt to guide search engine crawlers. If you want to tell crawlers which paths to skip, point them to your sitemap, and avoid syntax mistakes, this assembles a valid file from your rules that you save at your site root.
Benefits
- Builds a valid robots.txt in the standard format
- Adds User-agent, Allow and Disallow rules cleanly
- Includes an optional Sitemap directive for crawlers
- Generates the file locally with nothing uploaded
Practical use cases
- Blocking crawlers from an admin or staging path
- Pointing search engines to your sitemap location
- Creating a first robots.txt for a new site
- Tidying an existing file into valid syntax
Explore all SEO and Website Tools tools
Frequently asked questions
Where does robots.txt go?
At the root of your domain, for example https://example.com/robots.txt.
Can robots.txt keep a page private or secure?
No. robots.txt only asks compliant crawlers not to visit certain paths; it does not block access or hide content, and the file itself is publicly readable. For genuinely private pages use authentication or server-side restrictions rather than relying on a Disallow rule.
What does the Sitemap line in robots.txt do?
The Sitemap directive tells crawlers where to find your sitemap, helping them discover your pages more efficiently. It points to the sitemap's full URL and is optional but useful. You can include it alongside your Allow and Disallow rules in the same file.