Sign up free

Search Engine Spider Simulator

Fetch a page as Baiduspider, Googlebot or another crawler and compare it with what visitors get.

About Search Engine Spider Simulator

A spider simulator shows what your server sends to a crawler. Use it when a page ranks for odd keywords, when you suspect a hacked site is showing spam only to search engines, or when you want to confirm your content isn't hidden behind scripts. Choose the crawler (Baiduspider, Baiduspider mobile, Googlebot, Googlebot smartphone, Bingbot, 360Spider, Sogou spider or YandexBot), enter the URL and click Simulate.

The page is fetched as that crawler and, at the same time, as a normal browser. You see the status, final URL, title, description, keywords, robots meta, heading outline, the text the crawler reads with its word count, and a table of the links it finds. The two versions are compared by text similarity, title and links, which exposes cloaking, and the crawler's robots.txt verdict is shown. Requests come from our server with the crawler's user agent, not from the search engine's IP addresses, so sites that verify crawler IPs may answer differently. To check real crawler visits in your logs, use Verify Googlebot and Baiduspider IPs.

How to use Search Engine Spider Simulator

  1. 1
    Enter the URL

    Paste the address of the page you want to see through a crawler's eyes.

  2. 2
    Pick the crawler

    Choose Baiduspider, Googlebot, Bingbot or one of the other five.

  3. 3
    Click Simulate

    The page is fetched as the crawler and as a browser in parallel.

  4. 4
    Compare the two versions

    Check the similarity, title and links, then read the text the crawler got.

Why use Cubfile for this

  • Eight crawlers

    Baidu and Google on desktop and mobile, plus Bing, 360, Sogou and Yandex.

  • Side-by-side comparison

    Text similarity in percent, title and link count against a normal browser.

  • What the crawler reads

    Text, headings, links and word count, exactly as delivered.

  • robots.txt verdict

    Whether robots.txt lets that crawler fetch the page.

FAQ

Search Engine Spider Simulator: questions and answers

What is cloaking?
Showing search engines different content from what visitors see, which search engines treat as spam. Hacked sites often use it to hide injected links, so a big gap between the two versions is worth investigating.
Is this exactly what Googlebot sees?
It's what your server returns to a request with Googlebot's user agent. The real Googlebot also renders JavaScript and comes from Google's IP addresses, so sites that check IPs or build content with scripts can differ. Whether Google and Baidu may index the page at all is a question for the Indexability Checker.
Why is the crawler's text so short?
The content may load through JavaScript, be blocked for bots or sit behind a login. The simulator reads the HTML your server sends and warns when a page returns very little text.
Can it fetch the mobile version?
Yes. Choose Baiduspider (mobile) or Googlebot (smartphone) to receive the version your server gives phones.
Does the spider simulator use my daily tasks?
No. It's free, needs no sign-up and nothing is stored. An hourly allowance per visitor prevents abuse.
Share Search Engine Spider Simulator with a friendIt runs in any browser, and they can try it without signing up.

Related tools

SEO Broken Link CheckerCheck every link on a page and list the ones that are broken or redirected.
SEO SEO CheckerCheck a page on 40+ on-page and technical SEO points, with a fix for each problem.
SEO Indexability CheckerFind out whether a page can be indexed: status, robots, noindex and canonical in one check.
SEO Keyword Density CheckerSee which words and phrases a page repeats most, and how often.
SEO Meta Tag CheckerRead a page’s title, description, robots, canonical and social tags, with length checks.
SEO Robots.txt TesterRead a site’s robots.txt and test whether a URL is allowed for each crawler.
SEO AI Crawler CheckerSee which AI crawlers (GPTBot, ClaudeBot, Google-Extended…) your robots.txt lets in.
SEO Verify Googlebot and Baiduspider IPsCheck whether an IP in your logs really belongs to Google, Baidu or Bing.