SEO & Web
Robots.txt Generator
Build a correct robots.txt file from presets and rules — including crawl-delay, sitemap declaration, and per-bot policies — ready to upload.
User-agent: * Disallow:
robots.txt prevents crawling, not indexing. Upload to your site root exactly at /robots.txt. Per RFC 9309.
Disallow Is Not De-Indexing
Blocking a path in robots.txt prevents crawling, not indexing: a URL that is blocked but linked from elsewhere can still appear in results as a bare URL — the one case where you want meta robots noindex instead (which requires the page to be crawlable). The other classic mistake: Disallow: /tmp/ pattern confusion — the directive is prefix-based, so /tmp blocks /tmpfile.html too. And an empty robots.txt (or none) is a valid, crawl-everything configuration; do not add rules to feel complete.
How the Generator Works
- Choose the default policy: allow everything, block everything, or custom rules.
- Add per-bot rules with presets (Googlebot, Bingbot, GPTBot, etc.) including disallowed paths.
- Add your sitemap URL — declaring it in robots.txt is the standard discovery mechanism.
- Copy or download the generated robots.txt for your site root.
Example Output
Common Questions
Does robots.txt hide pages from Google?
No — it prevents crawling, not indexing. To truly exclude a page from the index use a noindex meta tag; robots.txt can even prevent Google from seeing that tag.
Where must robots.txt be placed?
Exactly at the site root (https://example.com/robots.txt). Subdirectory placements are ignored.
Related SEO & Web
XML Sitemap Generator
Turn a list of URLs into a valid XML sitemap with per-URL priority, changefreq, and lastmod — validated and downloadable.
Meta Tag Generator
Generate copy-paste-ready meta tags for your pages with a live Google-style search result preview and length guidance.
Canonical URL Generator
Normalize any URL into its canonical form and generate the matching <link rel="canonical"> tag with your preferred www/HTTPS/slash policy.