Robots.txt Generator
Build a robots.txt file with per-bot rules. Block AI crawlers, set crawl delays, define sitemaps.
robots.txt rules explained
The robots.txt file lives at the root of your domain and tells well-behaved web crawlers which pages they may or may not index. User-agent: * applies to all bots; specific agents like Googlebot or GPTBotoverride the wildcard. Disallow: /private/ blocks that path; Allow: / explicitly permits it. The Crawl-delay directive asks bots to wait N seconds between requests (Googlebot ignores it). The Sitemap: line helps Google discover all your pages. For generating a sitemap, see the Sitemap XML Generator.
Built and maintained by Meet Shah · Last updated
What this tool is used for
- Producing a robots.txt for a site that has none.
- Adding per-bot rules for AI crawlers alongside search crawlers.
- Declaring a sitemap location in the file.
- Producing a file to review before deploying it.
- Replacing an inherited file whose rules nobody documented.
How it works in practice
A worked example
A staging path and a checkout flow need keeping out of search results, and an AI crawler needs blocking from the whole site.
User-agent: * Disallow: /admin Disallow: /cart User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml
Googlebot /administrator/login DISALLOWED by "Disallow: /admin" (line 2, group *) Googlebot /blog/post ALLOWED — no rule matched GPTBot /blog/post DISALLOWED by "Disallow: /" (line 6, group GPTBot) GPTBot /cart DISALLOWED by "Disallow: /" (line 6, group GPTBot)
The second and third lines together are the thing to understand. A crawler obeys exactly one group — the most specific whose user-agent matches — and it does not inherit anything from the wildcard. So the AI crawler is blocked everywhere by its own single rule, and if you had instead added it as a second name under the shared group it would have received those two paths and nothing more. Getting this backwards is how a site ends up believing a bot is blocked while it crawls happily under the wildcard.
The edge case that catches people
The first line is the trap. A rule of slash-admin matched a request for slash-administrator, because these are prefix comparisons against the start of the path and not path-segment comparisons. Nothing about a directory boundary is implied, so blocking a short path blocks every longer path that begins with those characters — a rule meant for one admin area can quietly take out an unrelated section whose name merely starts the same way. Add the trailing slash when you mean the directory, and the longer path stops matching.
When not to use this tool
It is the wrong instrument for keeping a page out of search results, and the wrong instrument for privacy. A disallowed URL can still be indexed from other sites' links — with no snippet, because the crawler was never allowed to read the page — so the tag that actually removes a page is a noindex, which a blocked crawler will never see. And the file is public at a fixed, guessable location, so every path listed in it is advertised to anyone curious. Secrets need authentication; robots.txt is a request, honoured by the crawlers that choose to.
Frequently Asked Questions
- Does Disallow stop a page being indexed?
- No — it stops it being CRAWLED, which is a different thing. A blocked URL that other sites link to can still appear in results, described only by that anchor text, because the crawler was never allowed to read the page and find the `noindex` you put on it. Blocking and de-indexing need opposite instructions.
- How are conflicting rules resolved?
- By specificity, not order: the longest matching path wins, and `Allow` beats `Disallow` on a tie. So `Disallow: /admin/` with `Allow: /admin/public/` permits the subfolder, regardless of which line comes first. RFC 9309 finally wrote that rule down in 2022 after twenty years as a convention.
- Which user-agent line applies to a given crawler?
- Exactly one — the most specific matching group, and it does not inherit from `*`. If you write a `Googlebot` group, Googlebot ignores the `*` group entirely, including any Disallow lines you assumed still applied. This is the single most common robots.txt mistake.
- Where must the file be?
- At the root of the host: `https://example.com/robots.txt`, nowhere else, and it governs only that exact scheme, host and port. `https://www.example.com` and `https://example.com` are different origins with separate files, as is the same host on a different port.
- Can I use it to block AI crawlers?
- For the ones that honour it, yes — `GPTBot`, `Google-Extended`, `CCBot` and `ClaudeBot` are all documented tokens. It is entirely voluntary compliance, and a crawler that ignores robots.txt is unaffected by anything you write there, so it is a request rather than a control.
Common errors and gotchas
- Blocking `/_next/static/` or an equivalent asset path, which stops crawlers rendering any page — the single most damaging rule you can write.
- Using it to hide sensitive paths, when the file is public and therefore advertises them.
- Expecting a disallow to remove a page from the index, which needs `noindex` instead.
- Writing a rule under the wrong user-agent group, since only the most specific matching group applies.
- Blocking everything with a stray `Disallow: /` left over from a staging file.