DevCoreTools

robots.txt Tester

Paste robots.txt content and test whether a URL path is allowed or blocked for a specific crawler.

robots.txt input

Paste rules as text. This tester never fetches live robots.txt files or crawls websites.

Processed locally in your browser Limit robots.txt text. Your input is processed in your browser and is not uploaded or sent to a server. Avoid pasting production secrets, access tokens, private keys, passwords, or sensitive customer data unless you understand the risk.
0 input characters0 groups0 rules0 warnings

Test result

Matched group, longest rule, sitemaps, and crawl warnings.

Waiting for input/private/report.htmlGooglebot
robots.txt decisions appear here.

Load a sample or paste rules to check a path against a crawler user-agent, matching group, Allow/Disallow rule, and sitemap hints.

robots.txt testing guide

Validate crawler access rules before shipping changes to production robots.txt files.

User-agent groups

Crawlers choose the most specific matching User-agent group. A wildcard * group is the fallback when no named bot group matches.

Allow vs Disallow

Allow and Disallow rules are compared by the longest matching path. When rules tie, crawler behavior can vary, so avoid ambiguous duplicates.

Sitemap directives

Sitemap lines can live outside user-agent groups and should use absolute HTTP or HTTPS URLs that point to sitemap XML files.

Wildcard rules

This tester supports practical * wildcard matching and $ end anchors for path checks, which covers common robots.txt testing workflows.

Common mistakes

Empty user-agent groups, misspelled directives, malformed sitemap URLs, conflicting rules, and tokenized URLs are flagged before a file reaches production.

Local-only testing

The tool treats robots.txt and URL input as text. It does not request pages, fetch live robots.txt files, resolve domains, or contact external validators.

Keep working

View all HTML & Web tools
Open Graph / Meta Tag PreviewerPreview Open Graph, Twitter Card, and SEO meta tags from pasted HTML.Sitemap XML ValidatorValidate pasted sitemap XML and sitemap indexes locally with URL and field warnings.URL Parser / Query String ParserEncoding & DecodingParse full URLs, relative URLs, and query strings into structured parts.HTTP Headers Parser / FormatterAPI & Auth ToolsParse, format, inspect, and export HTTP headers locally with structured warnings.User Agent ParserAPI & Auth ToolsDecode browser, OS, device, engine, and bot hints from user agent strings locally.MIME Type LookupAPI & Auth ToolsLook up MIME types and file extensions locally with Content-Type header examples.HTML Previewer / Live HTML PreviewPreview HTML live in an isolated browser frame with safety warnings and export actions.

FAQ

Does this robots.txt Tester fetch my website?

No. It only parses pasted robots.txt text and the path you type. It does not fetch, crawl, resolve DNS, or call validators.

How does user-agent matching work?

The tester selects the most specific group whose user-agent token appears in the crawler name, with * used as the fallback group.

Which rule wins when Allow and Disallow both match?

The longest matching path rule wins. If there is still a tie, this tester prefers Allow, but you should remove ambiguous conflicts because crawlers can differ.

Can I test full URLs?

Yes. Full URLs are reduced to their path and query string for rule matching. Hosts are not fetched or validated.

Is Crawl-delay enforced?

No. Crawl-delay is parsed and displayed as a hint. Search engines and crawlers differ in whether they honor it.