Free Robots.txt Checker & Tester
Test any URL against your robots.txt, catch syntax errors, and see which search and AI crawlers can reach it. Googlebot, Bingbot, GPTBot, ClaudeBot and PerplexityBot are checked in one run. No signup.
Results appear here.
What is a robots.txt checker?
A robots.txt checker reads your robots.txt file and tells you which crawlers may fetch which URLs. It applies the matching rules crawlers use: the most specific user-agent group wins, the longest matching path rule wins, and Allow beats Disallow when two rules are the same length. Those rules are written down in RFC 9309, the robots exclusion standard.
The words checker, tester and validator get used interchangeably. In practice a validator checks the file's syntax, and a tester checks whether one URL is allowed for one crawler. This page does both, and it runs the test for every major crawler at once so you can compare Googlebot with the AI bots side by side.
The tool implements the RFC 9309 rules in your browser. It does not call Google's own parser, so treat the robots.txt report in Google Search Console as the final word for Googlebot, and use this page to find the problem and test the fix first.
How to test your robots.txt file
Five steps. The first result takes a few seconds.
- Enter the page URL you want to test. Use the exact URL, for example
https://yoursite.com/blog/post. To check whether a URL is blocked by robots.txt, this is the only field you must fill. - Pick a crawler. Googlebot is the default. Choose Bingbot, any AI crawler, or type a custom user-agent.
- Click Check robots.txt. The tool fetches
yoursite.com/robots.txt. To test a draft before it goes live, switch to Paste robots.txt and paste it in. - Read the verdict and the rule trace. The trace lists every rule that matched the URL and marks the one that decided the result.
- Edit the file in the box and check again. Repeat until the result is what you want, then publish the corrected file to your site root.
What the robots.txt validator checks
Every issue carries a line number and a suggested fix.
| Severity | What it catches | Why it matters |
|---|---|---|
| Critical | Disallow: / for all crawlers; rules placed before any User-agent line | Can stop every crawler from reaching your whole site, or leave rules that are silently ignored. |
| Warning | Robots.txt syntax errors such as a missing colon or a misspelled directive; wildcard errors such as $ in the middle of a path or a path without a leading slash; blocked CSS and JavaScript; noindex inside robots.txt; a relative Sitemap URL; conflicting Allow and Disallow rules; a file over 500 KiB | A rule may not work the way you wrote it, so a page you meant to hide stays crawlable or one you wanted indexed gets blocked. |
| Info | An empty Disallow; Crawl-delay; no Sitemap line; duplicate groups for one crawler; non-standard directives | Often intentional, but worth knowing. Google, for example, ignores Crawl-delay. |
Critical means the file is probably hurting crawling right now. Warning means a rule may not behave as written. Info is context.
Is your robots.txt blocking AI crawlers like GPTBot and ClaudeBot?
AI companies run more than one crawler, and each has its own robots.txt token. Blocking one does not block the others. That is why "block GPTBot" and "stay out of ChatGPT search" are two different decisions. The checker above shows all of them for any URL, so you can see what your file allows today.
| Crawler | Operator | What it does | Common choice |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Finds pages to show in ChatGPT search results | Allow, to be findable in ChatGPT search |
| ChatGPT-User | OpenAI | Opens a page when a person asks ChatGPT to read it | Allow |
| GPTBot | OpenAI | Collects content that may be used to train models | Your call, separate from search |
| Claude-SearchBot | Anthropic | Improves the quality of search results in Claude | Allow, to be findable in Claude |
| Claude-User | Anthropic | Opens a page when a person asks Claude about it | Allow |
| ClaudeBot | Anthropic | Collects content that may be used to train models | Your call |
| PerplexityBot | Perplexity | Indexes pages so Perplexity can cite them | Allow, to be cited |
| Perplexity-User | Perplexity | Opens a page when a person asks Perplexity about it | Allow |
| Google-Extended | A token that controls use of your content for Gemini training and grounding. It does not affect Google Search ranking | Your call | |
| Applebot-Extended | Apple | A token that controls use of your content for Apple's AI model training | Your call |
| CCBot | Common Crawl | Builds the open Common Crawl dataset that many models draw on | Your call |
Crawler names and behaviour change, and some user-triggered fetchers may not follow robots.txt in every case. Confirm details in each vendor's documentation before you rely on a rule.
Block training bots but stay citable
A common pattern: let AI search tools find and cite you, and opt out of model training.
# Let AI search tools find and cite this site User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: / # Opt out of model training User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
Allow GPTBot, ClaudeBot and PerplexityBot
If you want the widest AI visibility, name each crawler and allow it. This also protects you if a later User-agent: * block gets stricter.
User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: * Disallow: /admin/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
Brands that sell to buyers who ask AI tools for recommendations usually gain from being citable. Publishers who license their content often choose differently. See how AI visibility tools measure the result, or how PilotDeck tracks ChatGPT and Perplexity citations.
Common robots.txt mistakes and how to fix them
Leftover Disallow: / from staging
The most damaging one. A staging file gets copied to production and every crawler is told to stay out. Remove the line or scope it to the paths you mean, then test a live URL.
Blocked CSS and JavaScript files
Search engines render your pages. If /wp-includes/, /assets/ or your script folders are blocked, the rendered page can look broken to them. Allow the files a page needs to display.
Disallow not working
Usual causes: the rule sits in a different group than the one your crawler uses (a named group replaces *), a longer Allow rule overrides it, the path has the wrong case, or the page is already indexed through links. To remove a page from results, use noindex, not robots.txt.
Robots.txt wildcard errors
* matches any run of characters and $ anchors the end of the URL. A $ in the middle of a path is treated literally, and a path that starts with neither / nor * never matches. Major search engines support wildcards, but not every crawler does.
Missing or relative Sitemap line
The Sitemap directive needs a full URL, including https://. A path like /sitemap.xml is not a valid value. You can list more than one.
Noindex inside robots.txt
Google stopped supporting noindex in robots.txt in 2019. Use a meta robots tag or an X-Robots-Tag header instead, and leave the page crawlable so the tag can be read.
Robots.txt syntax errors
A missing colon, a misspelled directive such as Dissallow, or a rule before the first User-agent line. Crawlers skip what they cannot parse, so the rule you meant to write does nothing.
An empty Disallow
Disallow: with no path means "nothing is disallowed". It allows everything for that group. That is fine when intended and surprising when not.
Robots.txt syntax cheat sheet
The directives and patterns the checker understands.
| Directive | What it does | Example |
|---|---|---|
| User-agent | Starts a group and names the crawler it applies to. * means every crawler without its own group. | User-agent: Googlebot |
| Disallow | Blocks a path prefix for the group above it. | Disallow: /admin/ |
| Allow | Permits a path, usually to make an exception inside a blocked folder. | Allow: /admin/help/ |
| Sitemap | Gives the full URL of an XML sitemap. Applies to the whole file, not one group. | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay | Asks a crawler to wait between requests. Bing honours it. Google ignores it. | Crawl-delay: 5 |
| * (wildcard) | Matches any run of characters inside a path. | Disallow: /*?sessionid= |
| $ (end anchor) | Matches the end of the URL only. | Disallow: /*.pdf$ |
Precedence in one line: the crawler uses the most specific group that names it, then within that group the longest matching rule decides, and if an Allow and a Disallow tie on length, Allow wins.
Where to find and edit your robots.txt file
Your robots.txt file lives at the root of the host, always at https://yourdomain.com/robots.txt. Open that address in a browser to read it. A 404 means there is no file, and crawlers treat that as permission to crawl everything. Each subdomain needs its own file.
- WordPressServes a virtual file by default. Edit it from your SEO plugin, or upload a physical
robots.txt, which takes over. - ShopifyEdit the
robots.txt.liquidtemplate in your theme. - WebflowAdd rules under the site's SEO settings.
- Next.jsCreate
app/robots.ts, or place a static file atpublic/robots.txt. - Static sitesPut a plain-text
robots.txtin the folder that is served as your site root.
Sites made with AI builders such as Lovable, v0, Bolt and Framer can ship with a default or placeholder robots.txt. Run your live URL through the checker once after launch, and again after any redesign.
When to run a robots.txt test
- Before and after a launch or migrationStaging rules leaking into production is the classic way a site disappears from search.
- When Search Console says "Blocked by robots.txt"Paste the flagged URL to see the exact rule responsible, instead of guessing.
- Before changing your AI crawler policyCheck what GPTBot, ClaudeBot and PerplexityBot can reach now, then test the new file before it goes live.
- After a CMS, theme or plugin updateUpdates can rewrite a virtual robots.txt without warning.
- When you add or move a sitemapConfirm the Sitemap line is an absolute URL and that no rule blocks the sitemap path.
Try our other free SEO tools
Use Our Free SEO Tools Instantly - No Sign-Up Required.
SERP and rankings
- Google SERP checkerCheck where a page ranks in Google
- Bing SERP checkerCheck where a page ranks in Bing
- SEO title checkerTest title length and pixel width
- Meta description checkerSee where Google may cut it off
Technical SEO
- Robots.txt generatorSet crawl rules and block AI bots
- Sitemap checkerFind and audit XML sitemaps
- Sitemap XML generatorTurn a URL list into sitemap.xml
- Slug generatorMake short, clean URL slugs
AI search and GEO
- AI crawlability checkerTest AI bot access to your pages
- GEO content checkerCheck if a page can be quoted in AI answers
- LLMs.txt generatorDraft an llms.txt for your site
- AI visibility checkerSee if AI tools mention your brand
Keyword research
- AI keyword toolKeyword ideas grouped by intent
- LSI keywords generatorFind semantic variations
- Keyword density checkerMeasure how often terms repeat
- Semantic validatorCheck topical coverage
Content and writing
- Meta title generatorGenerate SEO title options
- Meta description generatorGenerate description options
- Content outline generatorBuild a SERP-informed outline
- Word counterCount words and reading time
Social and YouTube
- YouTube title generatorTitles that fit your topic
- YouTube SEO tags generatorTags viewers actually search for
- Reddit threads finderFind threads that rank and get cited
- LinkedIn post optimizerTighten a post for reach
LLM Trackers
PilotDeck covers every major LLM out there
ChatGPT trackerSee when ChatGPT mentions and recommends your brand.Explore ChatGPT visibility →
Perplexity trackerTrack whether Perplexity cites your pages in its answers.Explore Perplexity visibility →- AI citation analysisFind out which sources AI answers cite, and where you appear.Explore AI citation analysis →
Robots.txt checker FAQ
How do I check if my robots.txt file is working correctly?
Enter a page URL above and run the check. The tool reads your live robots.txt, lists syntax problems by severity, and shows whether the URL is allowed for the crawler you picked, along with the rule that decided it. Test one important page from each section of your site, not only the homepage.
How do I test if a URL is blocked by robots.txt?
Paste the full URL into the Page URL field and choose a crawler. The verdict says Allowed or Blocked, and the rule trace names the line responsible. If two rules match, the longer one wins, and Allow wins a tie.
What is the difference between robots.txt and noindex?
Robots.txt controls crawling: whether a bot may fetch a URL. Noindex controls indexing: whether a fetched page may appear in results. If you block a page in robots.txt, the crawler cannot see a noindex tag on it, and the URL can still be indexed if other sites link to it. To keep a page out of results, use noindex and leave it crawlable.
Should I block GPTBot and ClaudeBot?
It depends on what you want from AI answers. GPTBot and ClaudeBot are associated with model training, and blocking them opts your content out of that. They are separate from the search crawlers, OAI-SearchBot and Claude-SearchBot, that help AI tools find and cite pages. Brands that want to be recommended usually allow the search crawlers. Publishers who license content often block all of them.
Does blocking GPTBot remove my site from ChatGPT answers?
Not by itself. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search, so a rule for one does not control the other. Check both in the crawler table above and confirm current behaviour in OpenAI's documentation.
Where is the robots.txt file located?
At the root of your host: https://yourdomain.com/robots.txt. It must be at that exact location, and each subdomain has its own file. If the address returns a 404, the site has no robots.txt and crawlers may fetch everything.
How do I fix a page blocked by robots.txt?
First, run the URL through the checker to find the rule blocking it. Then remove that rule, narrow it (for example Disallow: /private/ instead of Disallow: /), or add a longer Allow rule for the path. Publish the updated file, retest the live URL, and request reindexing in Google Search Console.
What does Disallow: / mean?
Under User-agent: * it blocks every crawler from every path on the site. Use it on staging or development servers, not on a live site you want found. A copied staging file is the usual way it reaches production by accident.
Does Google support Crawl-delay?
No. Googlebot ignores the Crawl-delay directive. Bing and some other crawlers honour it. The checker lists it as Info for that reason.
Crawl access is one part of getting found
A clean robots.txt only opens the door. PilotDeck researches what your buyers search, publishes content against it every month, checks your technical foundation continuously, and tracks which AI engines name you.