Free Robots.txt Generator

A robots.txt generator turns a handful of plain choices into the text file that tells crawlers where they may go on your site. Instead of typing User-agent and Disallow lines by hand and hoping the syntax holds up, you pick options from a form and get a file in the format that Google, Bing and AI crawlers expect.

This robots.txt file generator covers more than the classic allow-or-refuse list. It names over 30 crawlers across search, AI and SEO tools, accepts custom rules for any user agent, and ships with templates for popular platforms. It also keeps your rules consistent: when one bot gets a group of its own, the folders you restricted for everyone are repeated inside that group, because a crawler that finds its own section stops reading the general one.

What is a robots.txt file?

It is a plain text file at the root of your domain that well-behaved bots read before they request anything else. The format belongs to the Robots Exclusion Protocol, which became an official internet standard, RFC 9309, in 2022. Each group in the file names one or more crawlers and then lists the paths they may or may not fetch.

What the file can and cannot do

Who uses a robots.txt generator

• SEO specialists and agencies set up crawl rules at launch and rebuild them after a site migration.
• Founders and site owners create a first robots.txt without learning the syntax.
• Developers keep a locked-down file for staging and a separate open file for production.
• Content and brand teams decide which AI tools may read and quote their pages.

Most people finish in under a minute. Here is how to create robots.txt online with the tool above.

1. Pick a starting point. Choose a blank file, Allow everything, Block everything, or a template for WordPress, WooCommerce, Blogger, Joomla, Drupal or Magento.
2. Set the default rule. Allow all is right for almost every live website. Block all suits staging servers and private projects.
3. Adjust individual crawlers. Set any search engine, AI crawler or SEO tool bot to Allowed or Refused. The AI shortcuts let you block model training while keeping AI search open in one click.
4. List restricted directories. Add each folder or page that no crawler should request, starting with a slash.
5. Add custom rules if you need them. Mix Allow and Disallow for any user agent, including wildcard patterns such as /*.pdf$.
6. Add your sitemap and an optional crawl delay. Use the full sitemap address, including https://.
7. Generate, review and save. Press Generate robots.txt, read the checks under the file, then copy it or download it. Upload it to your site's root folder and confirm it with the robots.txt checker.

A robots.txt file is a list of groups. Each group opens with one or more User-agent lines, followed by the rules for those crawlers. One directive goes on each line, and lines starting with # are comments.

Wildcards and matching

Paths are matched from the start of the URL and are case-sensitive, so /Blog/ and /blog/ are different rules. An asterisk stands for any run of characters and a dollar sign marks the end of the URL: Disallow: /*?sessionid= catches every URL carrying that parameter, and Disallow: /*.pdf$ catches every PDF. When an Allow and a Disallow rule both match, Google follows the longer, more specific one, and picks Allow if the two are equally long.

How groups decide which rules apply

A crawler obeys only the most specific group that names it. If Googlebot has its own group, it ignores everything under User-agent: *. This catches out many hand-written files, where a site blocks /admin/ for everyone, adds a Googlebot group for something else, and leaves the admin area open to Google. The generator avoids that by repeating your restricted directories in every named group that is allowed to crawl.

Each robots.txt example below is a complete file you can adapt. Swap in your own paths and sitemap address, or load the matching template in the generator to edit it there.

Allow all crawlers

The simplest valid file. Every crawler may fetch every URL.

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Disallow all crawlers

For staging sites and private projects only. Left on a live site, it hides every page from search.

User-agent: *
Disallow: /

Block one folder for everyone

User-agent: *
Disallow: /internal/
Disallow: /*?sort=

Refuse a single crawler

Everyone else keeps full access; only the named bot is turned away.

User-agent: *
Allow: /

User-agent: MJ12bot
Disallow: /

WordPress robots.txt

A robots.txt for WordPress keeps the dashboard out of reach but leaves the AJAX endpoint open, because themes and plugins use it on the public site. There is no need to block /wp-includes/, which holds scripts Google uses to render your pages.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://www.example.com/wp-sitemap.xml

Custom robots.txt for Blogger

Blogger's search and label pages create endless thin URLs, so a custom robots.txt for Blogger usually blocks /search while keeping the AdSense crawler fully open.

User-agent: Mediapartners-Google
Allow: /

User-agent: *
Disallow: /search
Allow: /

Sitemap: https://yourblog.blogspot.com/sitemap.xml

Online store with filters

Carts, accounts and sorted or filtered listings rarely belong in search. Blocking their URL patterns saves crawl budget for products and categories.

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?orderby=
Disallow: /*?filter_

Sitemap: https://www.example.com/sitemap_index.xml

AI companies now publish names for their crawlers, and those names fall into three jobs. Training crawlers collect pages that may be used to build future models. Search crawlers build the index that AI assistants search when they answer and cite sources. User-triggered fetchers open a page because a person asked the assistant to read it. Each job can be allowed or refused separately.

Stay citable, opt out of training

Many publishers want AI assistants to recommend and link to them without handing their pages to model training. The Block AI training, keep AI search shortcut in the generator builds exactly that file:

User-agent: *
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: Amazonbot
Disallow: /

What robots.txt will not do for AI

The file is a request. OpenAI and Perplexity both state that their user-triggered fetchers may not follow robots.txt, since a person asked for the page, and crawlers that do not identify themselves will not read it at all. If content must stay out of every AI tool, put it behind a login or block it at your firewall. To see whether AI assistants actually mention you once the doors are open, run the free AI visibility checker.

A leftover Disallow: / from staging

A development site is often blocked completely, and that file gets copied to production at launch. Check the live file the day you go live, and again after every redesign or migration.

Blocking CSS, JavaScript or images

Google renders pages much like a browser. If stylesheets and scripts are blocked, it may misread your layout and content. Keep folders such as /assets/ and /wp-includes/ crawlable.

Using robots.txt to hide pages

A disallowed URL can still be indexed if other pages link to it; it just shows without a description. Google stopped honouring noindex lines inside robots.txt in 2019. Use a noindex meta tag or an X-Robots-Tag header instead, and leave that page crawlable so the tag can be seen.

Forgetting that named groups override the default

Rules under User-agent: * do not carry over into a group that names a crawler. Repeat any restriction you still need, or let the generator do it for you.

A relative sitemap address

Sitemap: /sitemap.xml is not valid. The line needs the complete URL, starting with https://.

The wrong file name or location

The file must be called robots.txt in lowercase, saved as plain UTF-8 text, and served from the root of each host. A copy in a subfolder is ignored, and blog.example.com needs its own file.

Expecting Crawl-delay to slow Google

Googlebot ignores the line and adjusts its pace based on how quickly your server responds. Crawl-delay only affects crawlers that support it, such as Bingbot and YandexBot.

Upload the file to the top-level folder of your site so it opens at https://yourdomain.com/robots.txt. You do not need to submit it anywhere, because crawlers fetch it on their own. The exact steps depend on your platform.

• WordPressWordPress serves a virtual file by default. Edit it in your SEO plugin's file editor, or upload a real robots.txt to the root folder by FTP.
• BloggerSettings, then Crawlers and indexing. Switch on Enable custom robots.txt and paste the file.
• ShopifyShopify generates the file. To change it, edit the robots.txt.liquid template in your theme code.
• Wix and WebflowBoth have a robots.txt field in their SEO settings. Paste the generated text there.
• Next.jsPlace the file in the public folder, or return the rules from app/robots.ts.
• Static sites and serversSave the file in the web root next to your home page's index file.

Test before and after you publish

Open the live URL in a browser to confirm the new version is being served, then run important pages through the robots.txt checker to see which crawlers each rule lets in. Google Search Console's robots.txt report shows the version Google last fetched, any lines it could not parse, and lets you ask for a fresh fetch. Google reads up to 500 KiB of the file and usually refreshes its cached copy within a day.

When to review your robots.txt

• At launch, to confirm the staging block is gone.
• After a migration or redesign, when folder names and URL patterns change.
• When you add a section, such as a store, a forum or a members area.
• When a new AI crawler appears that you want to allow or refuse.
• When pages stop getting indexed without an obvious reason.

One file, or the whole site?

A robots.txt generator gets the rules right on day one. Keeping them right is harder, because plugin updates, new templates and site changes can block pages without anyone noticing. PilotDeck's site audit watches crawlability across your whole domain and flags problems as they appear, and you can start a free trial to run it on your site.