Check whether a URL is allowed or blocked by a robots.txt file.
Checked locally in your browser.
How do you check whether a URL is blocked by robots.txt?
For the matching user-agent group, every Allow and Disallow rule that matches the path is compared by length; the longest (most specific) rule wins, and Allow beats Disallow on a tie. A trailing $ anchors to the URL's end and * matches any characters. For example, with "Disallow: /private/" and "Allow: /private/public/", the path /private/public/page is allowed.
Understanding your result
Crawlers pick the rule group whose user-agent best matches, then apply the most specific matching path rule. A trailing $ anchors the match to the end of the URL, and * matches any run of characters.
Formula and method
For the matching user-agent group, every Allow and Disallow rule that matches the path is compared by length. The longest (most specific) rule wins, and Allow beats Disallow on a tie.
Assumptions and limitations
This tester checks only the robots.txt text you paste against a single URL, applying standard longest-match precedence. It does not fetch your live file, follow redirects, or account for meta robots tags, crawl-delay handling or the quirks of every individual crawler. Well-behaved bots obey these rules; it cannot stop crawlers that ignore robots.txt entirely.
Worked example
With “Disallow: /private/” and “Allow: /private/public/”, the path /private/public/page is allowed because the longer Allow rule wins.
How to use this tool
- Paste your robots.txt content.
- Enter the URL or path to test.
- Choose a user-agent and read the verdict.
Common mistakes to avoid
- Assuming the first matching rule wins — the most specific one does.
- Forgetting robots.txt controls crawling, not indexing — use noindex to keep a page out of results.
About the Robots.txt Tester
The Robots.txt Tester checks whether a given URL is allowed or blocked for a crawler, based on a robots.txt file you paste in. It applies the same precedence search engines use, and names the deciding rule.
Who should use this tool
SEOs, developers and site owners managing how search engines crawl their site.
Benefits
- Allowed or blocked verdict in one click.
- Honours wildcards (*) and end-anchors ($).
- Per-user-agent rule groups.
- Shows the exact deciding directive.
Practical use cases
- Checking a page is not accidentally blocked.
- Testing Disallow rules before publishing.
- Confirming Googlebot can reach key pages.
Explore all SEO and Website Tools tools
Frequently asked questions
Does robots.txt stop a page being indexed?
No. It controls crawling. A blocked URL can still appear in results if linked elsewhere; use a noindex meta tag or header to prevent indexing.
Which rule wins when several match?
The most specific (longest) matching path. If an Allow and a Disallow are the same length, Allow wins.
Do all search engines interpret robots.txt the same way?
Mostly, but not exactly. The longest-match, Allow-wins precedence used here follows the widely adopted convention, yet individual crawlers can differ on wildcard support, comment handling and edge cases. Use this as a reliable guide for major search engines, and check a specific bot's documentation when precise behaviour matters.