Robots.txt Tester

The Robots.txt Tester checks a list of paths against the rules in a robots.txt file and tells you, for a chosen crawler, whether each path would be allowed or blocked, along with the exact rule that decided the outcome. This lets you verify your crawl directives are doing what you intend before you deploy them, or diagnose why an important page is being kept out of search.

You paste your robots.txt content in one box and the paths you want to test in another, then pick the user-agent to evaluate. The tool parses the file the way crawlers do, selecting the rule group that applies to your chosen agent, matching wildcards and end-of-path anchors, and applying the longest-match rule to resolve conflicts between allow and disallow directives.

Everything is evaluated locally in your browser using JavaScript with no network requests, so your robots.txt and the paths you test are never uploaded. That makes it safe to check unpublished sites and confidential URL structures while getting an instant, rule-by-rule verdict for every path.

Features

• Tests many paths at once against a single robots.txt, returning a verdict for each.
• Evaluates rules for a specific user-agent, falling back to the wildcard group when needed.
• Shows the exact allow or disallow rule that determined each path's outcome.
• Supports wildcard patterns and end-of-path anchors the way real crawlers interpret them.
• Applies longest-match precedence so allow and disallow conflicts resolve correctly.
• Summarises how many paths are blocked versus allowed for a quick overview.
• Runs fully offline in your browser, keeping your robots.txt and URLs private.

How to use Robots.txt Tester

1. Paste the full contents of your robots.txt file into the first box.
2. Enter the paths you want to test in the second box, one per line, each starting with a slash.
3. Set the user-agent to the crawler you want to evaluate, such as Googlebot or the wildcard.
4. Read the table, where each row shows the path, whether it is allowed or blocked, and the matching rule.
5. Use the matching-rule column to understand and, if needed, correct a directive.
6. Adjust your robots.txt and retest until every path behaves the way you intend.

Benefits

• Developers confirm new crawl rules work before pushing robots.txt to production.
• SEO specialists diagnose why a page is blocked and identify the exact offending rule.
• Site owners verify that private sections are blocked and public ones remain crawlable.
• Agencies audit a client's robots.txt against a list of key URLs quickly.
• Teams catch overly broad disallow rules that unintentionally hide important pages.
• Because testing is local, unpublished sites and URL structures stay private.

Crawlers apply robots.txt rules by first selecting the group of directives that matches their user-agent, using the wildcard group only if no specific match exists. Within that group, the most specific rule wins, which for most modern crawlers means the longest matching path. This tool follows the same longest-match logic, so its verdicts closely mirror how Google and similar crawlers behave in practice.

Robots.txt patterns support an asterisk as a wildcard for any sequence of characters and a dollar sign to anchor a rule to the end of a path, which lets you target things like all URLs ending in a particular file type. Getting these patterns right is easy to misjudge, so testing real paths against them is the surest way to confirm that a rule blocks exactly what you expect and nothing more.

Keep in mind that robots.txt governs crawling rather than guaranteeing a page stays out of search, and that not every bot obeys the file. Use this tester to validate behaviour for compliant crawlers, and rely on authentication or noindex tags where appropriate. All evaluation happens in your browser with JavaScript and no network calls, so your robots.txt and tested URLs remain completely private.