Magictools · robots.txt
Robots.txt generator
Pick who can see your site, add any paths to hide from crawlers, and copy a robots.txt. Nothing is stored.
Choose who can see your site. The file appears after you pick one.
This file is a request, not a lock. A Disallow does not by itself remove a URL from Google.
Paths to allow or block
These rules apply to Google and to every crawler you do not name separately.
Shortcuts
Optional. * matches any sequence. $ anchors the end of the URL. The longest match wins, and Allow wins a tie.
Named bots
No extra bots named yet. Google and AI search follow the rules above.
Sitemaps
Enter a host to suggest a sitemap.
How it works
Choose who can see your site, add paths, then copy the file to your site root.
- Step 1
Choose who can see your site
Choose who can see your site before a file is generated.
- Step 2
Add paths and a sitemap
Add paths and a sitemap, or import a robots.txt you already have.
- Step 3
Copy the robots.txt
Copy or download the file and publish it at the host root.
More about this tool
Robots.txt Generator starts with a plain question: who can see your site? The recommended file stays visible to Google and to AI search. You can ask training crawlers to stay out, or block named AI crawlers, before any file is created. Then add Allow and Disallow paths, a sitemap, and named bots such as AdsBot-Google and Storebot-Google. The preview stays in your browser. Fetching an existing public robots.txt is optional and only happens after you opt in. The file is a voluntary hint: it does not control access, and Disallow does not by itself remove a URL from Google's index.
Frequently asked questions
Does this robots.txt block bots from my site?
No. robots.txt is a voluntary request. Well-behaved crawlers may follow it, and others may not. It is not access control, a firewall, or a privacy guarantee.
Will Disallow remove pages from Google?
No. Disallow asks Google not to crawl a URL. A URL can still be indexed without a snippet. Use noindex on a page Google is allowed to crawl when you want it kept out of the index.
Why do I have to choose who can see my site?
Visible to Google and AI search, visible to search while asking training bots to stay out, and blocking named AI crawlers are different decisions. The generator does not pick one for you.
Does User-agent: * cover AdsBot and Storebot-Google?
Storebot-Google uses its own group or the * group. AdsBot-Google and AdsBot-Google-Mobile ignore the * group, so name them when they should follow a rule. A Googlebot group does not apply to either AdsBot or Storebot-Google.
Is Google-Extended the same as Googlebot?
No. Google-Extended is a control token for Gemini training and grounding. It does not control Google Search and it does not change rankings. Googlebot is the search crawler.
Do you store the robots.txt I generate?
No. Generation, paste import, and the simulator run in your browser and are not stored. A public robots.txt is fetched only after you opt in, and that request is not kept as history.
Related tools
LLMs.txt Generator
Create llms.txt and llms-full.txt for AI agents. Free in your browser, with optional sitemap-first discovery and no signup.
Sitemap Generator
Discover public URLs or build XML sitemaps from your list. Free in your browser, with honest Google guidance and optional site discovery.
GEO & AI Crawler Audit
Audit AI crawler rules, llms.txt, agent files, structured data, FAQs, semantic HTML, and agent accessibility from source and rendered pages.
SEO, GEO & AI Audit
Audit one public page for SEO, GEO, AI understanding, content quality, entity clarity, schema, and citation readiness with evidence-first scoring and optional AI interpretation.
Robots.txt generator · Free, no account, no stored history