Robots.txt Generator — Manage Allow & Disallow Rules for AI Crawlers
Your robots.txt tells Google which pages to crawl and index — a wrong rule can hide your whole site from search. Generate a robots.txt file in your browser. Steps: pick user-agents, add Allow / Disallow paths and a Sitemap line, then copy the plain-text result with syntax rules and updated AI-crawler user-agents explained. Runs locally—nothing is uploaded. Example: blocking GPTBot while keeping Googlebot fully enabled.
Tick the crawlers to include, then add paths for each group.
How it works
Select the crawlers you target, add Allow and Disallow paths for each group, optionally add a Sitemap line, then Generate. The page assembles the plain-text file following the RFC 9309 layout: one User-agent line per group, its Allow/Disallow lines underneath, a blank line between groups, and the Sitemap line at the end. Copy it to the root of your site at /robots.txt.
Syntax rules that matter
robots.txt rules are path prefixes, not patterns, and a crawler applies the last matching group in the file. These are the rules this generator follows.
- Group order: a User-agent line starts a group that applies to that crawler; a blank line ends it. The last group matching a crawler wins.
- Prefix matching: Allow and Disallow match path prefixes, not substrings or regex. Only * and $ are special (RFC 9309).
- Disallow: / blocks that agent from the whole site. Disallow with an empty value allows everything for that agent.
- Sitemap: is an extension, not part of RFC 9309. It is case-insensitive and may appear anywhere, though it is usually placed at the end.
Example
Sample: Googlebot has Allow: / (fully enabled), GPTBot has Disallow: / (blocked), and a Sitemap line points to /sitemap.xml. The output mirrors this layout: one group per crawler, blank line between groups, sitemap last.
Good fits
- New sites: generate a starting robots.txt that keeps search engines enabled and hides private staging paths.
- AI crawler control: block GPTBot, ClaudeBot, Google-Extended, CCBot or PerplexityBot from training or summary extraction without touching Googlebot.
- Discovery: pair robots.txt with a Sitemap line so crawlers can find your sitemap URL.
Frequently asked questions
What happens if I write Disallow: / ?
It tells that crawler not to fetch any URL under the site root. If Googlebot gets Disallow: /, your pages can disappear from Google Search. For a staging path use Disallow: /private/ instead.
How do I block AI crawlers like GPTBot?
Create a group with the crawler’s User-agent (for example GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot) and add Disallow: /. Check the crawler’s official documentation, since AI crawlers update their user-agents and IP ranges from time to time.
Where should the Sitemap line go?
Position does not affect how search engines read it, but convention places it at the end of the file. The generator appends it last when you provide a URL.
Is my robots.txt uploaded anywhere?
No. The file is generated in your browser and never sent to a server. Close the tab and nothing is kept.
Related tools
Questions or feedback
Something unclear, broken, or missing? Draft a message below — we read every note about these tools.