Sign up free

Robots.txt Tester

Test whether a URL is allowed for 20 search and AI crawlers, and see the exact rule that decides.

About Robots.txt Tester

One wrong line in robots.txt can hide a whole site from search. Use this robots.txt tester before you upload new rules, when a page drops out of Google or Baidu, or when you want to know whether AI crawlers can reach your content. Enter a full page URL and click Test. The live robots.txt is fetched, or you can paste a draft and test it first.

Rules are read as Google documents them under RFC 9309: the * wildcard, the $ end anchor, the longest matching rule wins, Allow wins a tie, and the most specific user-agent group applies. The path is tested for 20 crawlers, from Googlebot, Bingbot and Baiduspider to 360Spider, Sogou, YisouSpider, YandexBot and PetalBot, plus AI crawlers such as GPTBot and ClaudeBot, and any crawler name you add. Each row shows allowed or blocked and the deciding rule, highlighted in the file. You're warned about Disallow: / for everyone, blocked CSS or JS, syntax errors, a missing Sitemap line, Crawl-delay, a Host directive, files over 500 KB and error status codes. Write new rules with the Robots.txt Generator.

How to use Robots.txt Tester

  1. 1
    Enter a page URL

    Use the full address of the page you want to test, not just the domain.

  2. 2
    Add a crawler or a draft

    Optionally type your own crawler name, or paste draft rules to test instead of the live file.

  3. 3
    Click Test

    Every crawler gets an allowed or blocked verdict for that path.

  4. 4
    Find the deciding line

    The rule that decides is highlighted in the file view, ready to adjust.

Why use Cubfile for this

  • Google's matching rules

    Wildcards, $ anchors, longest match and Allow on ties, as in RFC 9309.

  • 20 crawlers at once

    Search engines from Google to Baidu, Sogou and Yandex, plus the main AI crawlers.

  • Draft testing

    Paste rules and test them before they go live.

  • Clear warnings

    Blocking everything, blocked CSS or JS, syntax errors and a missing Sitemap line.

FAQ

Robots.txt Tester: questions and answers

Does Disallow in robots.txt stop a page from being indexed?
Not reliably. It stops crawling, but a blocked URL can still be indexed from links pointing to it, just without its content. To keep a page out of search results, allow crawling and add a noindex tag. The Indexability Checker then confirms whether Google and Baidu can index the page.
Does Google follow Crawl-delay?
No. Google ignores Crawl-delay, which is why the tester mentions it. Bing does support it.
What happens if robots.txt returns 404 or 500?
A 404 or other 4xx answer means there are no rules, so crawlers may fetch everything. A 5xx server error makes Google treat the whole site as blocked for a while, which is why the tester warns about it.
Can I test robots.txt before uploading it?
Yes. Open Test your own rules instead of the live robots.txt, paste the draft and click Test. The same URL and crawlers are checked against your text.
How do I block AI crawlers like GPTBot?
Add a group for each one, for example User-agent: GPTBot followed by Disallow: /. This tester checks GPTBot, ClaudeBot, Google-Extended, CCBot and PerplexityBot, and the AI Crawler Checker covers 25 AI crawlers.
Share Robots.txt Tester with a friendIt runs in any browser, and they can try it without signing up.

Related tools

SEO Broken Link CheckerCheck every link on a page and list the ones that are broken or redirected.
SEO SEO CheckerCheck a page on 40+ on-page and technical SEO points, with a fix for each problem.
SEO Indexability CheckerFind out whether a page can be indexed: status, robots, noindex and canonical in one check.
SEO Keyword Density CheckerSee which words and phrases a page repeats most, and how often.
SEO Meta Tag CheckerRead a page’s title, description, robots, canonical and social tags, with length checks.
SEO Search Engine Spider SimulatorSee a page the way Baiduspider or Googlebot does, and whether it differs from what visitors get.
SEO AI Crawler CheckerSee which AI crawlers (GPTBot, ClaudeBot, Google-Extended…) your robots.txt lets in.
SEO Verify Googlebot and Baiduspider IPsCheck whether an IP in your logs really belongs to Google, Baidu or Bing.