A robots.txt tester shows whether a crawler may fetch a URL on your site, based on the rules in your live robots.txt file. This one uses Google's matching logic and includes a generator. SEOs and developers use it to catch accidental blocks before they cost traffic.
- How to use the robots.txt tester
- How Google reads robots.txt rules
- Supported directives and what Google ignores
- Managing AI crawlers in robots.txt
- Robots.txt mistakes the tester catches
- FAQ
- Related tools
How to use the robots.txt tester
- Enter your domain. The tool fetches
/robots.txtfrom the site root and shows the file with line numbers so you can point to the exact rule. - Pick a crawler. Choose Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended or CCBot, or type a custom user-agent.
- Enter the path you want to test, for example
/blog/post-name/or/search?q=shoes. - Run the test. The result says allowed or blocked and highlights the line that decided it.
- Check the warnings. The tool lists your Sitemap lines and flags risky setups such as a blanket disallow or a missing sitemap.
- Need a file from scratch? Open the generator tab, set your rules, then copy or download the result.
The tester reads the file as it is live on your server. Edit your robots.txt, upload it, then run the test again to confirm the change took effect.
How Google reads robots.txt rules
The tester applies the same matching logic that Google documents in its robots.txt specification. Four points explain most results.
When several rules match a URL, Google uses the most specific one, which means the rule with the longest path.
If an Allow and a Disallow match with equal specificity, Google uses the least restrictive rule, so the URL is crawlable.
* matches any run of characters and $ anchors the end of the URL. Disallow: /*.pdf$ blocks every PDF.
Paths in rules are case-sensitive, so /Admin/ and /admin/ are different. User-agent names are not.
A short example shows the longest-match rule in action. Take these two lines:
Disallow: /shop/
Allow: /shop/sale/The URL /shop/sale/shoes/ matches both. The Allow path is longer, so the URL is crawlable. The URL /shop/boots/ matches only the Disallow, so it is blocked.
Think of the tester as a robots.txt checker and validator in one. Run it after every deploy, after a CMS or plugin update, and whenever you change URL structure, because each of those can silently rewrite the file.
Supported directives and what Google ignores
Google supports four fields: user-agent, allow, disallow and sitemap. Anything else is ignored. The tester warns you about unknown directives so you do not rely on rules that do nothing.
| Directive | Effect in Google |
|---|---|
| User-agent | Starts a group of rules for one crawler, or for all crawlers with * |
| Disallow / Allow | Blocks or permits crawling of matching paths |
| Sitemap | Gives the full URL of a sitemap. It is not tied to any user-agent group |
| Crawl-delay | Ignored by Google. Some other crawlers read it |
| Noindex | Not supported in robots.txt. Use a meta robots tag or X-Robots-Tag header |
The most expensive misunderstanding is the last row. Robots.txt controls crawling, not indexing. A URL you disallow can still appear in results if other pages link to it, usually without a description. To keep a page out of the index, let Google crawl it and serve a noindex rule, or put the page behind a login.
Managing AI crawlers in robots.txt
Many sites now decide which AI crawlers may collect their content. The tester has presets for GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended, and the generator can add Disallow groups for them with one toggle.
Two things to keep straight. First, a Disallow group only applies to crawlers that honor robots.txt. Reputable operators publish their user-agent names and say they respect it, but the file is a request, not a lock. Second, Google-Extended is a control token, not a separate crawler. Blocking it tells Google not to use your content for certain AI products, and it does not change how Googlebot crawls or ranks you in Search.
Review your own goals before blocking. Blocking AI crawlers can reduce how often your content is quoted in AI answers. Allowing them can feed models you do not control. It is a business choice, and the tester lets you check the result either way.
Robots.txt mistakes the tester catches
- Disallow all.
Disallow: /underUser-agent: *blocks the whole site. It is often left behind from a staging launch. - No sitemap line. Not an error, but a Sitemap line helps crawlers find your URLs faster.
- File too large. Google enforces a 500 KiB limit and ignores anything after it. Consolidate rules if you approach that size.
- Blocking CSS and JavaScript. Google needs those files to render pages. Blocking them can hurt how your content is understood.
- Unlisted rules. A rule placed before any
User-agentline applies to nobody. Keep every rule inside a group. - Wrong location. The file must sit at the root of the host, such as
https://example.com/robots.txt. Each subdomain and protocol needs its own. - Server errors. If robots.txt returns a 5xx error, Google first pauses crawling the site. A missing file (404) is treated as no restrictions.
Frequently asked questions
What is a robots.txt tester?
A robots.txt tester fetches your site's robots.txt file and shows whether a given crawler may access a given URL. It applies the matching rules the same way search engines do, then points to the exact line that allowed or blocked the path. Use it before and after every edit.
How do I check if a URL is blocked by robots.txt?
Enter your domain, choose a crawler such as Googlebot, type the path, and run the test. The result shows allowed or blocked and highlights the deciding rule. Test several paths, including parameter URLs and file types, because wildcard rules often catch more than you intended.
Does Disallow stop a page from being indexed?
No. Disallow stops crawling, not indexing. Google can still list a blocked URL if other sites link to it, though usually without a snippet. To remove a page from results, allow crawling and use a noindex meta tag or X-Robots-Tag header, or restrict access with a password.
Does Google obey crawl-delay in robots.txt?
No. Google ignores the crawl-delay directive. Some other crawlers read it, so it is harmless to keep, but it will not slow Googlebot. Google adjusts its own crawl rate automatically based on how your server responds, so if crawling strains your server, fix performance or use Search Console settings instead.
Should I block GPTBot and other AI crawlers?
It depends on your goals. Blocking prevents those crawlers from collecting your pages, which may reduce AI citations of your content. Allowing them does the opposite. Reputable crawlers honor robots.txt, but it is a request, not enforcement. Decide per bot, then test the file.
Where does the robots.txt file go?
It must live at the root of the host, for example https://example.com/robots.txt. Rules apply only to that exact protocol, host and port, so a subdomain needs its own file. The file must be UTF-8 plain text and, for Google, under 500 KiB.