Robots.txt Generator
Build a valid robots.txt with CMS presets, a sitemap field, and 2026 AI-crawler toggles (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and more). Generated and validated in your browser — nothing is uploaded.
Generator
CMS presetTraining bots default to Block. Search / user-initiated bots default to Allow. Uncheck to omit a bot from the file.
Live validator
How it works and when to use it
This tool builds a robots.txt from a CMS preset, extra paths, and per-bot Allow/Block toggles for 2026 AI crawlers. The preview updates as you change options. Copy the text or download robots.txt and place it at your domain root so it is reachable at https://yourdomain.com/robots.txt.
Use a generator when you launch a new site, change CMS, or want to opt out of AI training crawls without touching Googlebot. Presets are starting points — Shopify checkout paths and WordPress admin paths differ from a headless Next.js app. Review every Disallow before you upload.
Robots.txt is not authentication. Disallow only asks polite crawlers to skip a path. Private URLs still need login, noindex, or real access control. The in-browser validator checks syntax, a missing Sitemap, empty Disallow (allow-all), unknown directives, duplicate user-agents, and CSS/JS paths that can hurt rendering.
FAQ
What is a robots.txt file?
A robots.txt file lives at the root of your site (yoursite.com/robots.txt) and tells crawlers which paths they may request. It is a voluntary standard—well-behaved bots honor it. It is not a security control and does not hide URLs from users or determined scrapers.
Does this robots.txt generator upload my site data?
No. The generator and validator run entirely in your browser. Your sitemap URL, custom paths, and preview never leave this page. There is no server-side processing and nothing is stored on a remote server.
Should I block GPTBot and other AI crawlers in 2026?
It depends on your goals. Blocking GPTBot, Google-Extended, Bytespider, and similar training crawlers can limit use of your content for model training. It will not stop Google Search—Googlebot is separate from Google-Extended. Search-oriented agents such as OAI-SearchBot or ChatGPT-User are optional: allow them if you want citations in AI answers.
What is the difference between Googlebot and Google-Extended?
Googlebot crawls for Google Search and should almost never be blocked site-wide. Google-Extended controls whether your content may be used to train Gemini and related Google generative products. This tool lists Google-Extended, not Googlebot.
Do I need a Sitemap line in robots.txt?
A Sitemap: line is optional but recommended. It does not replace Google Search Console submission. Use the full HTTPS URL to your XML sitemap or sitemap index. This validator warns if the line is missing.
Will blocking AI bots hurt my Google rankings?
Blocking training or AI-answer crawlers does not, by itself, change Google Search rankings. Do not Disallow Googlebot or block CSS and JavaScript that Google needs to render the page. The validator flags common render-blocking paths.
How do I use the WordPress robots.txt preset?
Choose WordPress to prefill Disallow /wp-admin/ and /wp-includes/ plus Allow /wp-admin/admin-ajax.php. Review /wp-includes/—Google prefers access to CSS and JS there. Upload the file to your site root, or paste the rules into an SEO plugin if WordPress is already serving robots.txt.
What does an empty Disallow mean?
Disallow: with no path means allow everything for that user-agent. Disallow: / blocks the whole site for that bot. The validator warns on an empty Disallow so you do not publish an accidental allow-all rule.