PopularHot
Meta Tag Generator
Meta Tag Generator that outputs clean, copy-ready code for better rankings.
SEOOpen Tool
Robots.txt Generator
Generate a robots.txt file — set whether crawlers and AI bots may access your site and point them at your sitemap, then copy the result to your site root.
Loading Robots.txt Generator…
Your browser is preparing the tool. It runs 100% locally.
Quick Answer
Toggle whether to allow all crawlers and whether to explicitly allow AI bots (GPTBot, ClaudeBot, PerplexityBot and Google-Extended), add your sitemap URL, and the tool writes a robots.txt you save at your site's root. robots.txt is the standard, RFC 9309 way to guide crawlers — but it is advisory: well-behaved bots obey it, and blocking a page does not guarantee it stays out of search. It runs in your browser.
What the Robots.txt Generator Does
It produces a simple, valid robots.txt: a User-agent star block set to Allow / or Disallow / depending on the first toggle, optional explicit Allow blocks for the major AI crawlers, and a Sitemap line pointing at your XML sitemap. Save it as robots.txt at the root of your domain.
The AI toggle is about being explicit. When it is on, the file spells out that GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and Google-Extended (Google's AI training) are allowed. It is a clear, public statement of your stance for the crawlers that honour it.
How It Works
The file is plain text with a fixed grammar: User-agent lines name a crawler, and Allow or Disallow lines set what it may fetch. A path of / means the whole site. The tool assembles those lines from your toggles and appends the Sitemap directive so crawlers can find your URL list.
robots.txt belongs at exactly one place — yourdomain.com/robots.txt — and crawlers read it before fetching other pages. It became an official internet standard (RFC 9309) in 2022, formalising rules that had been a de-facto convention since 1994.
Generation algorithm
- Set the crawler rule. Allow all sets User-agent star to Allow /; turning it off sets Disallow / to discourage all crawling.
- Add AI bots (optional). If enabled, append explicit Allow blocks for GPTBot, ClaudeBot, PerplexityBot and Google-Extended.
- Add the sitemap. Append a Sitemap line pointing at your XML sitemap URL.
- Output live. Show the finished robots.txt to copy to your site root; it updates as you toggle.
What it outputs
User-agent: *
Allow: / (or Disallow: / when 'allow all' is off)
(optional) Allow blocks for GPTBot, ClaudeBot, PerplexityBot, Google-Extended
Sitemap: your-sitemap-URLAllow-all or disallow-all only — no per-path rules or crawl-delay. Turning the AI toggle OFF removes the explicit allow blocks; it does not add a Disallow, so those bots still fall under the User-agent star rule.
Technical Details
| Inputs | sitemap URL, allow-all toggle, allow-AI toggle |
|---|---|
| Output | User-agent star rule, optional AI allow blocks, Sitemap line |
| Granularity | allow-all or disallow-all (no per-path rules) |
| AI bots named | GPTBot, ClaudeBot, PerplexityBot, Google-Extended |
| File location | yourdomain.com/robots.txt (root only) |
| Where it runs | In your browser — nothing uploaded |
Standards & references
- Robots Exclusion Protocol (RFC 9309) — the robots.txt standard, made an official IETF specification in 2022; compliant crawlers must follow the rules, but it is guidance, not access control.
- Disallow is not noindex — a blocked URL can still be indexed if linked elsewhere (often with no snippet). To keep a page out of search, use a noindex meta tag and leave it crawlable.
- AI crawler tokens — GPTBot (OpenAI), Google-Extended (Google AI), ClaudeBot (Anthropic) and PerplexityBot are the user-agents this file names; some AI crawlers ignore robots.txt entirely.
Accuracy & Limitations
robots.txt is advisory. Reputable crawlers (Google, Bing) obey it, but it does not enforce anything — some bots, including certain AI scrapers, ignore it outright. It guides behaviour; it is not a lock.
Disallow is not the same as noindex. A page blocked in robots.txt can still appear in search if other sites link to it — Google just shows it without a description. To truly keep a page out of search, use a noindex meta tag and leave the page crawlable (do not block it here). Note that Google has ignored noindex inside robots.txt since 2019.
Turning off 'Allow AI bots' does not block them. It only removes the explicit Allow blocks; those crawlers then fall under User-agent star, so they are still allowed if 'allow all' is on. To actually block an AI bot you would add a Disallow rule for it, which this tool does not generate.
It only does whole-site allow or disallow. There is no way here to block a single folder, add a crawl-delay, or set different rules per bot beyond the AI block — edit the file by hand for that.
It must live at the root as yourdomain.com/robots.txt. A robots.txt in a subfolder is ignored, and the Sitemap line should be a full absolute URL.
Real-World Use Cases
New site setup
Create a first robots.txt that allows crawling and points at your sitemap.
Declaring an AI stance
Explicitly allow (or, after editing, block) AI training crawlers.
Staging sites
Switch to Disallow / to discourage crawling of a site that is not ready.
Pointing to the sitemap
Add the Sitemap directive so crawlers discover your URL list.
When to use it — and when not to
Good for
- A simple allow-all robots.txt with a sitemap line
- Explicitly allowing the major AI crawlers
- Discouraging crawling of a staging site
- Learning robots.txt structure
Not the best choice for
- Blocking specific folders or URLs (no per-path rules)
- Keeping a page out of search (use noindex)
- Actually blocking AI bots (it only allows them)
- Crawl-delay or per-bot fine-tuning
Need to keep a page out of Google? Use a noindex meta tag, not robots.txt. Need per-folder rules or to block a bot? Edit the file by hand after generating the base.
Frequently Asked Questions
What is robots.txt for?
It is a file at your site's root that tells crawlers which parts of your site they may fetch. It is the standard way (RFC 9309) to guide search and other bots.
Does blocking a page in robots.txt hide it from Google?
Not reliably. A blocked page can still be indexed if other sites link to it, shown without a snippet. To keep it out of search, use a noindex meta tag and leave the page crawlable.
Does turning off 'Allow AI bots' block them?
No. It only removes the explicit allow lines; those bots then match the User-agent star rule. To block one, add a Disallow rule for it by hand.
Which AI bots does it name?
GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and Google-Extended (Google's AI training crawler).
Will every bot obey this file?
No. Reputable crawlers do, but robots.txt is advisory — some scrapers, including certain AI crawlers, ignore it entirely.
Where do I put the file?
At the root: yourdomain.com/robots.txt. It is ignored anywhere else, and there is one per domain.
Can I block just one folder?
Not with this tool — it does whole-site allow or disallow only. Add a Disallow line for the folder by hand for that.
What does the Sitemap line do?
It tells crawlers where your XML sitemap is, helping them discover all your URLs. Use the full absolute URL.
Is Disallow / a good way to take a site offline from search?
It discourages crawling, but pages already known can linger in the index. For a private site, use authentication or noindex, not just robots.txt.
Does Google read noindex in robots.txt?
No. Google dropped support for noindex directives in robots.txt in 2019. Use a noindex meta tag on the page instead.
Is robots.txt case-sensitive?
The paths are. /Folder/ and /folder/ are different, so match your real URL casing.
Is anything uploaded?
No. The file is built in your browser; save it and upload it to your own site root.
References
Builds a valid robots.txt with a sitemap line and optional AI-bot allow blocks; it is explicit that robots.txt is advisory, that Disallow is not noindex, and that turning the AI toggle off does not block those bots.
Related Tools
Popular
Keyword Density Checker
Keyword Density Checker online — accurate, standards-compliant and free for SEO work.
SEOOpen Tool
Popular
UTM Link Builder
UTM Link Builder online — accurate, standards-compliant and free for SEO work.
SEOOpen Tool
Trending
Open Graph Generator
Open Graph Generator that outputs clean, copy-ready code for better rankings.
SEOOpen Tool
Trending
SERP Snippet Preview
SERP Snippet Preview online — accurate, standards-compliant and free for SEO work.
SEOOpen Tool
Editor's Pick
Schema Markup Generator
Schema Markup Generator online — accurate, standards-compliant and free for SEO work.
SEOOpen Tool
Ready to try the Robots.txt Generator?
It is free, private and runs entirely in your browser — no sign-up, no uploads, no limits.