Fix Guide

How to Add a robots.txt File

Add the file that tells crawlers what they can crawl and where your sitemap lives.

Easy Medium Impact 10 min

Why It Matters

robots.txt guides crawlers — keeping them out of admin, cart and search pages so crawl budget goes to content that matters, and pointing them at your sitemap. Note it controls crawling, not indexing.

How to Fix

Create /robots.txt

Place a plain-text file at the site root. Allow crawling of content, disallow private/utility paths.

Reference your sitemap

Add a Sitemap: line with the absolute URL of your XML sitemap.

Code Fix

text

Before (broken)

# https://example.com/robots.txt → 404

After (fixed)

User-agent: *
Allow: /
Disallow: /cart
Disallow: /admin
Disallow: /*?q=

Sitemap: https://example.com/sitemap.xml

Platform-Specific Instructions

WordPress

Yoast/Rank Math include a robots.txt editor; otherwise drop the file in the site root via FTP.

Next.js

Add app/robots.ts exporting the rules + sitemap reference.

Check with our tool

Robots.txt Generator

Want to find all issues at once?

Run a free audit and get a prioritized fix list with auto-generated code.

Run a free audit

How to Add a robots.txt File — Fix Guide | SlapMyWeb