Search Engines Blocked Checker
Find out which search crawlers your robots.txt lets in — and which one it is quietly turning away.
- 100% free
- No sign-up
What is Search Engines Blocked Checker?
Blocking a search engine is almost never a decision someone remembers making.
It arrives in a robots.txt copied from a staging server, in a security plugin’s "block bad bots" default, or in a rule written to stop one crawler that catches nine. This checker reads the live robots.txt for the URL you give it and resolves the rules the way the crawlers do, for ten search crawlers at once. Two details separate it from most robots.txt checkers. It matches wildcard patterns properly, so a rule like Disallow: /*.pdf$ is reported as blocking your PDFs rather than nothing at all. And it knows that Google’s ad crawlers, AdsBot-Google and Mediapartners-Google, do not obey User-agent: * — a group has to name them — so it will not tell you they are blocked on a site where they are not, or clear them on a site where they are.
How to use Search Engines Blocked Checker
- 1
Enter the page you care about
Rules are per path, so the homepage and /checkout can have different answers. Give the URL you actually want in search.
- 2
Start with the Googlebot row
If that says Blocked, nothing else on the page matters — it cannot be fetched, so it cannot rank.
- 3
Read "No rule" as allowed
robots.txt is permissive by default. No rule means nothing is stopping the crawler, not that something is missing.
- 4
Check the blocks were intentional
Blocking Baiduspider on a site with no Chinese market is a choice. Blocking Bingbot takes you out of Bing, Copilot, DuckDuckGo and Yahoo at once, which rarely is.
Everything this tool
does for you
- Ten search crawlers checked against the live robots.txt
- Correct wildcard (*) and end-anchor ($) pattern matching
- Models the AdsBot and Mediapartners wildcard exemption
- Flags a 5xx robots.txt, which stops crawling site-wide
- Detects a robots.txt that is really an HTML soft 404
- Reads the robots meta tag and X-Robots-Tag header alongside
Further reading
Related free tools
Frequently asked questions
Beyond this tool
See the full picture for your website.
Enter your URL for a free audit across eight scored areas. See prioritised issues and clear next steps in one report.
Free to start · No signup required
- Checks ten search crawlers, not just Googlebot
- Understands wildcard and $ rules, which prefix matching gets wrong
- Knows that Google’s ad crawlers ignore User-agent: *
- Catches a noindex that a Disallow stops Google ever reading
- Free, no account, no daily limit