Robots.txt Generator
Build a clean robots.txt in seconds — block paths, add your sitemap, and optionally keep AI training crawlers out, all in one click.
# robots.txt — generated with mysitegptassistant.com/tools
User-agent: *
Disallow: /admin
Disallow: /dashboard
Disallow: /api
Sitemap: https://example.com/sitemap.xml
Save this as robots.txt at the root of your site (/robots.txt). It tells crawlers which pages they may fetch.
What this robots.txt generator does
Build a clean robots.txt in seconds: allow or disallow specific paths, add your sitemap, set a crawl delay, and — with one toggle — block the major AI training and answer crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and more) while staying open to Google and Bing.
That last control matters more every month: plenty of sites now want to be found in classic search but kept out of model training data. This generator makes that a checkbox instead of a research project.
How to install and verify it
- Copy the generated file and upload it to your domain root as
robots.txt. - Confirm it loads at
yoursite.com/robots.txt. - Then audit what crawlers will actually reach with the sitemap analyzer, and if you want AI assistants to read the right pages, add an llms.txt file alongside it.
Robots.txt generator FAQ
robots.txt is a plain-text file at the root of your domain (yoursite.com/robots.txt) that tells web crawlers which paths they may or may not request. It's the standard way to keep bots out of admin, cart, or API routes and to point them at your sitemap.
Add Disallow rules for the AI crawler user-agents — GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot, Bytespider and others. Toggle 'block AI crawlers' here and the generator emits those blocks for you, so you can stay open to classic search engines while keeping content out of model training.
Yes. A Sitemap: line pointing at yoursite.com/sitemap.xml helps search engines discover all your URLs. It's independent of the Allow/Disallow rules and is a simple, high-value addition.
No. Well-behaved crawlers respect it, but it's a request, not access control — and disallowed URLs can still be discovered via links. For anything truly private, use authentication or a noindex header, not robots.txt alone.
Yes. Choose your rules, toggle AI-crawler blocking, add your sitemap, and copy the file — free, no sign-up, generated in your browser.