Robots Meta Tag Generator
Build HTML meta robots tags and X-Robots-Tag headers for search engines and social bots.
Target Bot: robotsDirectives: index, follow
Configure crawler directives:
Target bot (name attribute)
HTML meta tag
HTTP header equivalent
Robots meta tags vs robots.txt
The <meta name="robots"> tag tells search engine crawlers how to handle a specific page. Unlike robots.txt (which controls crawling), meta robots tags control indexing and snippet behaviour — they are read after the page is fetched. Use noindex to remove a page from search results, and nofollow to prevent link equity from passing to external links. You can also target individual bots like googlebot or bingbot.
For whole-site crawl rules, use the Robots.txt Generator. For HTTP security headers, see the Security Header Generator.
Built and maintained by Meet Shah · Last updated
What this tool is used for
- Producing a noindex tag for a page that should not appear in results.
- Adding nofollow or noarchive directives deliberately rather than by copy-paste.
- Getting the header form for a non-HTML file such as a PDF.
- Producing per-crawler directives for a specific bot.
- Generating a tag to compare against what a page currently emits.
Frequently Asked Questions
- What is the difference between noindex and nofollow?
- `noindex` keeps the page out of the index; `nofollow` tells the crawler not to pass signals through the links on it. They are independent — a page can be indexed with nofollowed links, or excluded while still passing link equity onward.
- Why does a noindex page have to stay crawlable?
- Because the crawler has to fetch the page to see the tag. Blocking it in robots.txt means the instruction is never read, and a URL blocked from crawling can still be indexed from external links alone. To remove a page, allow the crawl and use `noindex`.
- What do noarchive, nosnippet and noimageindex do?
- `noarchive` suppresses the cached copy, `nosnippet` suppresses the description text in results, and `noimageindex` keeps the page's images out of image search. They are refinements of how an INDEXED page appears rather than whether it is indexed.
- Can I target one crawler?
- Yes — use the bot's token as the `name` attribute instead of `robots`: `<meta name="googlebot" content="noindex">`. A bot-specific tag overrides the generic one for that bot only, which is how a page is hidden from one engine and left in another.
- Meta tag or X-Robots-Tag header?
- The header does the same job and works on non-HTML resources — a PDF or an image cannot carry a meta tag. Use the meta tag for pages and the header for files, and never both with conflicting values, because the most restrictive one wins.
Common errors and gotchas
- Combining noindex with a robots.txt disallow, where the crawler cannot fetch the page to see the noindex.
- Applying noindex site-wide from a staging config and shipping it, which deindexes everything.
- Expecting a noindex to remove a page instantly, when it takes a recrawl.
- Using nofollow on internal links as a crawl-budget tool, which is not how it now behaves.
- Setting the tag with JavaScript, where the crawler may never see it.
Related Web & SEO tools
Meta Tag Generator
Build SEO title, description, and viewport meta tags.
Open Graph Generator
Generate Open Graph and Twitter Card meta tags.
Robots.txt Generator
Build a robots.txt file with per-bot rules. Block AI crawlers, set crawl delays, define sitemaps.
UTM Campaign URL Builder
Append UTM parameters to build trackable campaign URLs.
Hreflang Tag Generator
Generate hreflang link tags for multilingual SEO.
Sitemap XML Generator
Build an XML sitemap from a list of URLs.
SERP Snippet Preview
Preview how your page title and meta description appear in Google search.
Meta Tag Analyzer
Paste a page's HTML head to audit its title, meta, and social tags.