Technical SEO

Robots.txt

A file that tells search engines which parts of your site they can and cannot crawl.

robots.txt is a file at the root of a domain telling crawlers which paths they may request. It controls crawling, not indexing — a URL blocked here can still appear in results if other pages link to it, and because it cannot be fetched, any noindex on it is never read. Use it for crawl control and noindex for exclusion.

Definition

Robots.txt is a text file at the root of your website (/robots.txt) that tells search engine crawlers which pages or sections they are allowed or disallowed from crawling. It uses the Robots Exclusion Protocol standard.

Important: robots.txt only controls crawling, not indexing. A disallowed page can still appear in search results if other pages link to it. Use noindex meta tag to prevent indexing.

How common is this?

Across every site SlapMyWeb has audited, 80% serve a robots.txt — which leaves 20% with no robots.txt.

Measured from SlapMyWeb audits of live public URLs. The full aggregate is published as a CC-BY-4.0 dataset.

Why It Matters

Proper robots.txt prevents search engines from wasting crawl budget on irrelevant pages (admin panels, API endpoints, search result pages). It is essential for crawl budget optimization on large sites.

Example

User-agent: *
Allow: /
Disallow: /dashboard
Disallow: /admin
Disallow: /api

Sitemap: https://slapmyweb.com/sitemap.xml

The robots.txt and noindex contradiction

This is the most consequential misunderstanding in technical SEO, and it is worth stating plainly: robots.txt controls crawling; the robots meta tag controls indexing. They are not two ways of doing the same thing.

If you block a URL in robots.txt, crawlers do not fetch it — so they never see the noindex on it. The page can remain in the index indefinitely, listed on the strength of links pointing at it, showing no description because the content was never read.

The correct sequence to remove a page: allow crawling, serve noindex, wait for it to drop out, and only then block it in robots.txt if you want to save the crawl budget. Doing it in the other order is how pages get stuck.

AI crawlers are decided here

GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended are all controlled by robots.txt, and this is where a site decides whether it can be cited by answer engines at all.

Our own study of 300 top sites found 41% blocking at least one major AI crawler — and in most cases it appeared to be an inherited default rather than a decision anyone made. If AI referral traffic matters to you, this file is the first place to check, because no amount of content work compensates for a crawler that is not allowed in.

How SlapMyWeb checks this

The audit fetches /robots.txt, parses it, and reports whether it exists, whether it declares a sitemap, and — the check that catches real damage — whether it blocks paths that also carry noindex, which is the combination that leaves a URL permanently stuck in the index. It also reports which AI crawlers are allowed or denied. The matching fix guide is how to add a robots.txt file.

Related Terms

XML Sitemap Crawl Budget Technical SEO

Related Tools

Robots.txt Generator

Know the term.
Check your own site.

A free audit tells you whether this is currently costing you score points — and exactly what to change.

Run a free audit

Free foreverNo signupResults in 30s