Skip to main content
SlapMyWeb
Slap a site

SEO fix

How to Add a robots.txt File

Tell crawlers what they may request, and point them at your sitemap.

Difficulty
Easy
Impact
Medium
Time
15 min

Before

GET /robots.txt -> 404

After

User-agent: *
Allow: /
Disallow: /admin
Disallow: /api
Disallow: /search

Sitemap: https://example.com/sitemap.xml
Before and after code for: How to Add a robots.txt File

A missing robots.txt is not fatal, crawlers assume everything is allowed, but you lose the ability to steer crawling and to advertise your sitemap. Adding one lets you keep crawlers out of admin paths and internal search results, and it is where AI crawler access is granted or denied.

Why it matters

It controls crawling, not indexing: a URL blocked here can still appear in results, and because it cannot be fetched, any noindex on it is never read. Blocking a page you wanted removed is the classic mistake this causes.

How to fix it

  1. Serve it at the root

    It must be at /robots.txt on each host. A file at /public/robots.txt in your app is fine as long as that is where it resolves.

  2. Start permissive

    Allow everything, then disallow the specific paths that should not be crawled: admin, API endpoints, internal search results.

  3. Declare your sitemap

    A Sitemap: line with the absolute URL is how crawlers that did not come from Search Console find it.

  4. Decide about AI crawlers

    GPTBot, ClaudeBot, PerplexityBot and Google-Extended are named here. Around 41% of top sites block at least one, often without realising.

The code

Beforetext
GET /robots.txt -> 404
Aftertext
User-agent: *
Allow: /
Disallow: /admin
Disallow: /api
Disallow: /search

Sitemap: https://example.com/sitemap.xml

On your platform

WordPress

WordPress serves a virtual robots.txt if no real file exists. Adding a physical file overrides it entirely, so include everything you need.

Next.js

Use app/robots.ts to generate it, or a static file in public/. Do not ship both: the static file wins and the generated one is silently ignored.

Sources

More fixes

SEO

How to Remove an Accidental noindex

Find and remove a stray noindex that is keeping an important page out of Google entirely.

SEO

How to Fix Broken Internal Links (404s)

Find and repair internal links that point to missing pages so you stop leaking link equity.

SEO

How to Fix Redirect Chains

Collapse multi-hop redirects (A→B→C) into a single hop to the final URL.

Fixed that one.
Now find the rest.

A free audit ranks every issue on your site by the score points it costs, and links each one straight to its guide.

Run a free audit
Free foreverNo signupLive scan progress