API & Web

robots.txt Validator and Tester

Checks a robots.txt line by line and tests a path against it using the real longest-match rule.

Loading the tool…

Processing happens locally in your browser. What you paste or load is processed by this page and is not uploaded to a server. Nothing is stored unless you use a control that says it stores something, and you can clear anything this site has kept from the privacy page.

How to use this tool

  1. Paste the whole robots.txt file into the first box.
  2. Put the path you want to test in the second box, starting with a slash.
  3. Set the user agent to test as, or leave it at * for the general case.
  4. Select Check, then read the last row for the verdict and the exact rule that decided it.

What robots.txt validator does

Every line is reported with what it does, including the ones that quietly do nothing: a rule sitting above any User-agent line belongs to no group, an unknown field is ignored by every major crawler, and Crawl-delay is honoured by Bing and Yandex but not by Google. An empty Disallow is explained rather than flagged, because it is the standard way of saying there are no restrictions.

Put a path in the second box and it is tested against the file properly. A named user-agent group wins outright over the wildcard group rather than adding to it, the longest matching pattern wins between rules, and Allow beats Disallow on a tie — which is what Google and Bing both do, and is not what most people assume. The * and trailing $ wildcards are supported.

Frequently asked questions

The way crawlers actually do: a group naming your user agent wins outright, and the wildcard group is used only when no named group matches. Groups do not combine. That is the rule people most often get wrong — adding a Googlebot group means Googlebot stops reading the * group entirely.

The longest matching pattern wins, and on a tie Allow wins. So Disallow: /admin/ with Allow: /admin/public/ leaves the public subtree crawlable, because the Allow pattern is longer. This is what Google and Bing both do; the original 1994 standard said nothing about it.

No, and this is the most expensive misunderstanding in the file. It asks crawlers not to fetch a page. A page that is linked from elsewhere can still be listed without ever being fetched — and because the crawler cannot read it, a noindex tag on that page will never be seen. To keep something out of an index, allow the crawl and use noindex, or require authentication.

An asterisk matching any run of characters, and a trailing dollar anchoring the match at the end of the path. Both are extensions to the original standard that the major crawlers support. Nothing else is a wildcard — a question mark in a pattern is a literal question mark.