Skip to main content
SlapMyWeb
Slap a site
Fix Guide

How to Add a robots.txt File

Tell crawlers what they may request, and point them at your sitemap.

Easy Medium Impact 15 min

A missing robots.txt is not fatal — crawlers assume everything is allowed — but you lose the ability to steer crawling and to advertise your sitemap. Adding one lets you keep crawlers out of admin paths and internal search results, and it is where AI crawler access is granted or denied.

Why It Matters

It controls crawling, not indexing: a URL blocked here can still appear in results, and because it cannot be fetched, any noindex on it is never read. Blocking a page you wanted removed is the classic mistake this causes.

How to Fix

1

Serve it at the root

It must be at /robots.txt on each host. A file at /public/robots.txt in your app is fine as long as that is where it resolves.
2

Start permissive

Allow everything, then disallow the specific paths that should not be crawled: admin, API endpoints, internal search results.
3

Declare your sitemap

A Sitemap: line with the absolute URL is how crawlers that did not come from Search Console find it.
4

Decide about AI crawlers

GPTBot, ClaudeBot, PerplexityBot and Google-Extended are named here. Around 41% of top sites block at least one, often without realising.

Code Fix

text
Before (broken)
GET /robots.txt -> 404
After (fixed)
User-agent: *
Allow: /
Disallow: /admin
Disallow: /api
Disallow: /search

Sitemap: https://example.com/sitemap.xml

Platform-Specific Instructions

WordPress

WordPress serves a virtual robots.txt if no real file exists. Adding a physical file overrides it entirely, so include everything you need.

Next.js

Use app/robots.ts to generate it, or a static file in public/. Do not ship both — the static file wins and the generated one is silently ignored.

Fixed that one.
Now find the rest.

A free audit ranks every issue on your site by the score points it costs — and links each one straight to its guide.

Run a free audit
Free foreverNo signupResults in 30s