omid57

Robots.txt Generator

Create a valid robots.txt file: block folders, add exceptions and your sitemap, and optionally opt out of AI crawlers. Free.

Updated October 9, 2026 By

Runs in your browser Free · No sign-up

Loading tool…

How to use the Robots.txt Generator

  1. Choose whether crawlers are allowed in general, or start from a preset such as WordPress.
  2. Enter the paths to block, any exceptions, your sitemap URL and whether AI crawlers should be blocked.
  3. Copy or download the file and upload it as robots.txt to the main folder of your website.

About this tool

The robots.txt file tells search engines which parts of your website they may crawl. Choose what to block, add your sitemap and download a correctly formatted file that you upload to the root folder of your domain.

  • Private: your data is processed on your device and never uploaded.
  • Free with no account, limits or watermarks.
  • Fast: results appear instantly, even offline once the page has loaded.

What the Robots.txt Generator does

This generator creates a robots.txt file — the small text file in the root of every website that tells crawlers such as Googlebot and Bingbot which areas they may visit. You decide whether the site is open to crawling, list the folders or URL patterns that should be blocked, add exceptions, and include the address of your XML sitemap so search engines find all your pages.

Presets for WordPress and online shops fill in common rules, such as blocking the admin area while keeping admin-ajax.php reachable, or keeping cart, checkout and filter URLs out of the crawl. An optional switch adds rules for crawlers that AI companies use to collect training data or power AI search, such as GPTBot, ClaudeBot, Google-Extended and CCBot.

The file is built in your browser and can be copied or downloaded as robots.txt, ready to upload.

When to use it

  • New websites

    Publish a clean robots.txt with your sitemap from day one so search engines discover your pages faster.

  • WordPress and shops

    Keep admin, cart, checkout and filter pages from wasting crawl budget.

  • Staging and test sites

    Block a development copy completely so it does not compete with your live site.

  • AI opt-out

    Tell AI crawlers that respect robots.txt not to use your content.

Tips for the best results

  • Never block your CSS and JavaScript folders; Google needs them to render and understand your pages.
  • Paths are case-sensitive and match from the start: “/admin” also blocks “/administrator”. End folders with a slash.
  • Use * as a wildcard and $ for the end of a URL, for example “/*.pdf$” to block all PDF files.
  • robots.txt is public. Do not list secret folders in it — protect them with a password instead.
  • After uploading, check the file in Google Search Console under Settings › robots.txt.

How the Robots.txt Generator works

The generator writes one group for all crawlers (User-agent: *) with Allow and Disallow lines in the format defined by the Robots Exclusion Protocol (RFC 9309), which Google, Bing and other major search engines follow. Paths without a leading slash get one automatically.

If you block AI crawlers, a second group lists their user agents with “Disallow: /”. Crawlers use the most specific group that matches their name, so search engines stay unaffected. Sitemap lines are added at the end and may point to several sitemaps.

Crawl-delay is included only when you enter a value; Google ignores it, but Bing and some other crawlers respect it.

Frequently asked questions

Where does robots.txt go?

It must be in the root of your domain, for example https://www.example.com/robots.txt. Files in subfolders are ignored.

Does robots.txt remove pages from Google?

No. It stops crawling, but a blocked page can still appear in results if other sites link to it. Use a noindex meta tag on the page to keep it out of the index.

Do all bots respect robots.txt?

Reputable search engines and most AI companies do. Malicious bots and scrapers may ignore it, so it is not a security measure.

Will blocking AI crawlers affect my Google rankings?

No. Google-Extended only controls the use of your content for Google’s AI models; normal Googlebot crawling for Search is not affected.