Robots.txt Tester
Test any URL and user-agent against robots.txt rules. Parses allow, disallow, and crawl-delay directives.
Status: ALLOWEDParsed Rules: 0Crawl Delay: None
Parsed rules (0)
User-agent: *
Disallow: /admin/
Disallow: /private/
Crawl-delay: 1
User-agent: Googlebot
Allow: /
Crawl-delay: 0.5
Sitemaps (1)
https://example.com/sitemap.xml
How robots.txt works
The robots.txt file sits at the root of your domain and tells crawlers which pages they may or may not access (per the Robots Exclusion Protocol, RFC 9309). User-agent blocks are matched by substring. The most specific Allow/Disallow rule (by path length) takes precedence. A missing or empty robots.txt means all crawlers may access all pages.Crawl-delay asks bots to wait N seconds between requests.
Built and maintained by Meet Shah · Last updated
What this tool is used for
- Checking whether a specific URL is blocked before wondering why it is not indexed.
- Testing a rule change against real paths before deploying it.
- Working out which of several rules actually applies to a path.
- Confirming that an asset directory is crawlable, which is a documented cause of ranking loss.
- Checking how a rule behaves for one user-agent versus another.
Frequently Asked Questions
- What does robots.txt actually control?
- CRAWLING, not indexing — and the distinction causes real problems. A blocked page can still appear in search results if other sites link to it; Google just cannot see its content, so it shows a bare URL.
- How do I actually keep a page out of the index?
- With a noindex meta tag or X-Robots-Tag header — and then you must NOT block it in robots.txt, because a blocked crawler never sees the noindex. The two directives working against each other is a very common mistake.
- How are conflicting rules resolved?
- By specificity: the longest matching path wins, regardless of order. So Disallow: /admin/ and Allow: /admin/public/ combine correctly. Google also supports * and $ wildcards, which the original standard did not.
- Which user-agent group applies?
- Exactly one — the most specific match. A crawler matching a named group ignores the * group ENTIRELY, so rules you assumed were global do not apply to it. Directives must be repeated per group.
- Is robots.txt enforceable?
- No. It is advisory, and only well-behaved crawlers honour it. Anything genuinely sensitive needs authentication — and listing a secret path in robots.txt publicises it, since the file itself is public.
- Does robots.txt apply across subdomains and protocols?
- No. It governs exactly its own origin, so `blog.example.com` needs its own file and does not inherit the one at `example.com`. HTTP and HTTPS are separate origins too, though a redirect to the canonical one usually makes that moot.
Common errors and gotchas
- Assuming a blocked page is de-indexed, when robots.txt prevents crawling rather than indexing.
- Blocking CSS or JavaScript directories, which stops a crawler rendering the page at all.
- Expecting the longest rule to win by position, when the most specific match applies.
- Forgetting that a user-agent block matches the most specific group only, not every group.
- Using robots.txt to hide something sensitive, when the file itself is public and lists the paths.
Related Web & SEO tools
Meta Tag Generator
Build SEO title, description, and viewport meta tags.
Open Graph Generator
Generate Open Graph and Twitter Card meta tags.
Robots.txt Generator
Build a robots.txt file with per-bot rules. Block AI crawlers, set crawl delays, define sitemaps.
UTM Campaign URL Builder
Append UTM parameters to build trackable campaign URLs.
Hreflang Tag Generator
Generate hreflang link tags for multilingual SEO.
Sitemap XML Generator
Build an XML sitemap from a list of URLs.
SERP Snippet Preview
Preview how your page title and meta description appear in Google search.
Meta Tag Analyzer
Paste a page's HTML head to audit its title, meta, and social tags.