What is the difference between crawlable and indexable?

Crawlable means a search engine is allowed to fetch the URL. Indexable means it is allowed to list it. They are set in different places — robots.txt for the first, a robots meta tag or X-Robots-Tag header for the second — and a page needs both to appear in search. A page can be crawlable and deliberately not indexable, which is how you keep a thank-you page out of results.

Why does my page pass every check and still not rank?

Because crawlability is the floor, not the ceiling. These nine checks confirm nothing is stopping a search engine reading and listing the page. Whether it then ranks depends on the content, the competition and the links pointing at it — none of which is a technical setting.

Does a redirect hurt crawlability?

One hop costs almost nothing. The cost is in chains: every extra hop is another request a crawler spends on this URL instead of a new one, and on a large site that adds up to pages that never get crawled at all. Point the first URL straight at the last one.

Is a missing sitemap a real problem?

It is a missed opportunity rather than a fault. Search engines find pages by following links, so a well-linked site gets crawled without one. A sitemap matters most for pages that are new, deeply buried, or not linked from anywhere obvious — and the Sitemap: line in robots.txt is how a crawler that has never seen your site finds it.

My canonical points at a different URL. Is that wrong?

Only if it was not intended. A canonical pointing elsewhere is correct on a duplicate — a print view, a parameterised variant, a syndicated copy — and tells search engines to consolidate signals on the original. It is wrong when it is on the original, because the page is then asking not to be indexed in favour of something else.

SEO

Website Crawlability Test

Nine checks between a crawler and your page — the response, the redirects, robots.txt, the robots tag and the canonical.

100% free
No sign-up
No limits
Instant results

The short version

What is Website Crawlability Test?

Crawlability is not one setting.

It is a sequence, and a page drops out of search the moment any step in it fails: the URL has to respond, the redirects have to end somewhere, robots.txt has to permit the fetch, the robots directives have to permit the listing, and the canonical has to point at this page rather than somewhere else. Each of those has its own dedicated checker; this one walks the whole path in a single pass so you can see which step is the one that broke. The checks are ordered the way a crawler meets them, so the first failure is usually the one to fix — the ones after it are often symptoms rather than causes.

Step by step

How to use Website Crawlability Test

1
Test a page, not just the domain
Crawlability is decided per URL. A homepage can be perfect while every product page is blocked by one rule.
2
Fix the first failure first
The checks run in the order a crawler meets them. A 404 makes everything below it meaningless; a blocked fetch makes the robots tag unreadable.
3
Read the warnings as questions
A cross-domain canonical and a missing sitemap are both deliberate on some sites. The tool flags them so you confirm, not so you change them.
4
Re-test after deploying
Most crawlability failures arrive with a deploy — a staging robots.txt, a CDN rule, a framework default. A second test takes seconds and catches it before a crawl does.

What's inside

Everything this tool
does for you

Response status, including the difference between 4xx and 5xx
Full redirect chain with a hop count
robots.txt presence, status and soft-404 detection
Googlebot’s verdict on the exact path you entered
Sitemap declarations read from robots.txt
Robots meta tag and X-Robots-Tag header
Canonical tag resolved against the final URL

Keep going

Related free tools

All SEO tools

Meta Tag Generator

Create perfect SEO meta tags with OG + Twitter cards. Live character counters.

SEO

Meta Tag Analyzer

Analyze any URL's meta tags, OG tags, Twitter cards and SEO elements with scoring.

SEO

Open Graph Tag Generator

Generate Open Graph meta tags with live Facebook and LinkedIn preview.

SEO

Open Graph Checker

Check any URL's Open Graph tags and preview how links appear on social media.

SEO

Twitter Card Generator

Generate Twitter Card meta tags with live preview for summary and large image cards.

SEO

Robots.txt Generator

Build a robots.txt file with a visual editor. Add rules for any user agent.

SEO

Questions

Frequently asked questions

This tool fixed one thing.
Now fix the whole site.

Our scanner runs 240+ checks across 10 audit areas, ranks every issue by the score points it costs, and hands you the code to paste.

Run a free audit

240+ checksResults in under a minuteNo signup

What this tool does

Tests the whole path a crawler takes, not one signal
Catches the 5xx robots.txt that stops crawling site-wide
Separates "cannot be fetched" from "will not be listed"
Checks the canonical actually points at this page
Free, no account, no daily limit

Want the whole picture? Run 240+ checks on your site — free, under a minute, no signup.Run a free audit