Sign up free

Robots.txt Generator

Build a robots.txt with rules for search engines and AI crawlers, ready to upload to your site.

More tools
More generators301 Redirect GeneratorHtpasswd GeneratorHreflang Tag GeneratorUTM Link Builderllms.txt Generator

About Robots.txt Generator

A robots.txt generator helps when a new site goes live, when a staging copy shouldn't be crawled, or when you'd rather AI companies didn't train on your articles. Start with the default for all crawlers, allow all or block all, then list the folders to keep crawlers out of, one per line, such as /admin/ or /cart/. Add your sitemap URLs and, if a bot is straining your server, a crawl delay. The file updates as you type. If you don't have a sitemap yet, the XML Sitemap Generator can build one.

Each crawler also has its own switch. The search engines are Googlebot, Bingbot, Baiduspider, 360Spider, Sogou, YisouSpider (Shenma), YandexBot and DuckDuckBot. The AI crawlers are GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot, Bytespider, Applebot-Extended, meta-externalagent and Amazonbot, and one preset blocks AI training crawlers in a click. Copy the result or download robots.txt for your site root, then test real URLs with the Robots.txt Tester. Everything runs in your browser.

How to use Robots.txt Generator

  1. 1
    Choose the default rule

    Allow all crawlers or block all of them as the starting point.

  2. 2
    Add folders and sitemaps

    List the folders to block, one per line, then add your sitemap URLs and an optional crawl delay.

  3. 3
    Set crawlers one by one

    Switch search engines and AI crawlers on or off, or use the preset that blocks AI training crawlers.

  4. 4
    Download robots.txt

    Copy the text or download the file, upload it to your site root and check it with the Robots.txt Tester.

Why use Cubfile for this

  • Chinese search engines included

    Baiduspider, 360Spider, Sogou and YisouSpider sit next to Googlebot, Bingbot, YandexBot and DuckDuckBot.

  • Nine AI crawlers

    GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot, Bytespider, Applebot-Extended, meta-externalagent and Amazonbot.

  • One-click AI preset

    Block the crawlers that collect AI training data without writing each rule by hand.

  • Live and private

    The file is built in your browser as you type, with nothing uploaded and no daily limit.

FAQ

Robots.txt Generator: questions and answers

Where do I put the robots.txt file?
In the root of your site, so it opens at https://example.com/robots.txt. Crawlers don't look for it in subfolders, and each subdomain needs its own file.
Does robots.txt keep a page out of Google?
Not reliably. It stops crawling, but a blocked URL can still be listed if other pages link to it. To keep a page out of results, let it be crawled and add a noindex robots meta tag. The Meta Tag Generator writes that noindex tag for you.
How do I block GPTBot and other AI crawlers?
Switch off the AI crawlers you don't want, or use the preset that blocks AI training crawlers. Blocking Google-Extended doesn't change how your site appears in Google Search.
Does Google follow Crawl-delay?
No. Googlebot ignores it, while Bingbot respects it. Leave it empty unless a crawler is slowing your server down.
Is the robots.txt generator free?
Yes. It runs entirely in your browser, needs no sign-up, doesn't use any daily tasks and keeps working offline once the page has loaded.
Share Robots.txt Generator with a friendIt runs in any browser, and they can try it without signing up.

Related tools

XML XML Sitemap GeneratorCrawl your site or paste a list of URLs to build an XML sitemap.
HTML Meta Tag GeneratorWrite title, description and robots tags with live length checks and a preview.
JSON Schema Markup GeneratorGenerate JSON-LD for FAQ, articles, products, organizations and more.
HTML Open Graph GeneratorMake Open Graph and Twitter Card tags and preview the share card.
CONF 301 Redirect GeneratorGet redirect rules for Apache, Nginx, IIS, PHP and HTML.
URL UTM Link BuilderAdd UTM tags to links so you can track campaigns in analytics.
HTML Hreflang Tag GeneratorBuild hreflang tags for every language version of a page.
CONF Htpasswd GeneratorCreate .htpasswd lines for Apache and Nginx basic authentication.