Toolkeel

Robots.txt tester and validator

Find out whether a page is blocked for a search engine, and which line of robots.txt blocks it. The tool also catches the typos and unsupported rules that cause indexing surprises.

  • Free
  • Private
  • No sign-up
Open yoursite.com/robots.txt in a new tab, copy everything, and paste it here.
Stays on this device. Network requests since you started: 0 How to check this yourself
  1. Your text and dataRead from your device
  2. Worked on in this tabBy code running in your browser
  3. Ready to copyNothing is stored

Never sent to a server. See how to check

How to use it

  1. Paste the contents of your robots.txt file.

  2. Enter a URL or path and choose a crawler.

  3. Select "Test URL" to see whether it is allowed or blocked, and by which line.

About this tool

The file robots.txt tells crawlers which parts of a site they may fetch. This tester applies the rules of RFC 9309, the standard Google and Bing follow:

  • A crawler uses the group naming it most specifically. If no group names it, it uses the User-agent: * group. It does not combine the two.
  • Within that group, the longest matching path wins. If an Allow and a Disallow rule are equally long, Allow wins.
  • * matches any characters and $ marks the end of the URL.

Blocking is not hiding. A URL blocked in robots.txt can still appear in search results if other sites link to it. To keep a page out of Google, allow crawling and add a noindex meta tag instead.

Google crawlers such as Googlebot-Image fall back to the Googlebot group when no group names them. This tool tests the exact token you choose, so test both if it matters.

Good to know

  • Tests pasted text. The tool does not download robots.txt from your site.
  • Crawler-specific fallbacks (for example Googlebot-Image using the Googlebot group) are not applied automatically.
  • Crawl-delay is reported but not simulated.

Frequently asked questions

Why is my page blocked when I only disallowed a folder?

Rules match the start of the path, so "Disallow: /blog" also blocks /blog-news and /blogging. Add a trailing slash ("/blog/") to block only the folder.

Does Google support Noindex in robots.txt?

No. Google stopped supporting it in 2019. Use a meta robots tag or an X-Robots-Tag header.

How do I block AI crawlers?

Add a group for each crawler, for example "User-agent: GPTBot" followed by "Disallow: /". Test each one here. Not every AI company publishes or follows a crawler token.

You might also need

Last updated 2026-10-20 by the Toolkeel team. How we build and check our tools.