SEO

Robots.txt Generator

Generate a properly formatted robots.txt file to control how search engine crawlers access your site.

100% free
No sign-up
No limits
Instant results

Quick Presets

User-Agent

Disallow

Crawl Delay (seconds, optional)

Sitemap URL

robots.txt

User-agent: *
Disallow:

What this tool does

Control which bots can crawl your site
Block AI bots from scraping your content
Set crawl delays to protect server resources
Include sitemap URL for better discovery
One-click presets for common configurations

Want the full picture?

This tool fixes one thing. Our scanner runs 240+ checks across 10 SEO pillars and hands you a prioritised fix list — plus a savage AI roast.

Run a free audit

What is Robots.txt Generator?

A robots.txt file is a plain text file placed at the root of your website that tells search engine crawlers which pages or sections they are allowed or disallowed from accessing. The Robots.txt Generator by SlapMyWeb lets you visually build this file without memorizing the syntax. You can add multiple user-agent blocks, set allow/disallow rules per path, configure crawl delays, and include your sitemap URL -- all with one-click presets for common configurations like blocking AI bots (GPTBot, ClaudeBot, Bytespider, CCBot) from scraping your content.

How to use Robots.txt Generator

1
Choose a preset or start from scratch
Click "Allow All", "Block All", or "Block AI Bots" for instant configurations, or build custom rules manually.
2
Configure user-agent rules
Set the user-agent name (e.g., Googlebot, GPTBot, or * for all bots), then add Allow and Disallow paths for each agent.
3
Add sitemap and crawl delay
Enter your sitemap URL so crawlers discover all pages. Optionally set a crawl delay to protect your server from aggressive bots.
4
Copy and upload
Copy the generated robots.txt content and save it as a file named robots.txt in your website root directory.

Features

Visual editor with multiple user-agent blocks
One-click presets: Allow All, Block All, Block AI Bots
Block GPTBot, ChatGPT-User, ClaudeBot, Bytespider, CCBot
Add unlimited Allow and Disallow path rules
Sitemap URL inclusion for crawler discovery
Configurable crawl delay per user-agent
Real-time preview as you edit
One-click copy to clipboard

Related tools

Meta Tag Generator

Create perfect SEO meta tags with OG + Twitter cards. Live character counters.

Open tool

Meta Tag Analyzer

Analyze any URL's meta tags, OG tags, Twitter cards and SEO elements with scoring.

Open tool

Open Graph Tag Generator

Generate Open Graph meta tags with live Facebook and LinkedIn preview.

Open tool

Open Graph Checker

Check any URL's Open Graph tags and preview how links appear on social media.

Open tool

Twitter Card Generator

Generate Twitter Card meta tags with live preview for summary and large image cards.

Open tool

Robots.txt Tester

Paste your robots.txt and test if a URL is crawlable by any user agent.

Open tool

See all SEO tools

Frequently asked questions

Where do I place the robots.txt file?

The robots.txt file must be placed at the root of your domain, accessible at https://yourdomain.com/robots.txt. Search engine crawlers look for it at this exact location before crawling your site.

Does robots.txt block pages from appearing in Google?

No. Robots.txt prevents crawling, not indexing. If other pages link to a disallowed URL, Google may still index it (showing a URL-only result). To prevent indexing, use a noindex meta tag or X-Robots-Tag HTTP header.

Should I block AI bots like GPTBot and ClaudeBot?

That depends on your goals. If you don't want AI companies scraping your content for training data, blocking GPTBot, ChatGPT-User, ClaudeBot, Bytespider, and CCBot via robots.txt is the standard approach. However, this won't prevent content already scraped.

What does the * wildcard mean in User-agent?

User-agent: * matches all crawlers that don't have a more specific rule. It's the default catch-all. Specific user-agent blocks (like Googlebot) take priority over the wildcard for that particular bot.

Can I set different rules for different bots?

Yes. Add separate rule blocks for each bot. For example, you might allow Googlebot to crawl everything while blocking Bingbot from /private/. Each bot follows its own most-specific matching block.

What is Crawl-delay and should I use it?

Crawl-delay tells bots to wait a specified number of seconds between requests. Google ignores this directive (use Search Console instead), but Bing, Yandex, and others respect it. Use it if aggressive crawling is straining your server.

What is Robots.txt Generator?

How to use Robots.txt Generator

Choose a preset or start from scratch

Click "Allow All", "Block All", or "Block AI Bots" for instant configurations, or build custom rules manually.

Configure user-agent rules

Set the user-agent name (e.g., Googlebot, GPTBot, or * for all bots), then add Allow and Disallow paths for each agent.

Add sitemap and crawl delay

Enter your sitemap URL so crawlers discover all pages. Optionally set a crawl delay to protect your server from aggressive bots.

Copy and upload

Copy the generated robots.txt content and save it as a file named robots.txt in your website root directory.

Features

Visual editor with multiple user-agent blocks

One-click presets: Allow All, Block All, Block AI Bots

Block GPTBot, ChatGPT-User, ClaudeBot, Bytespider, CCBot

Add unlimited Allow and Disallow path rules

Sitemap URL inclusion for crawler discovery

Configurable crawl delay per user-agent

Real-time preview as you edit

One-click copy to clipboard

Frequently asked questions

Where do I place the robots.txt file?

The robots.txt file must be placed at the root of your domain, accessible at https://yourdomain.com/robots.txt. Search engine crawlers look for it at this exact location before crawling your site.

Does robots.txt block pages from appearing in Google?

Should I block AI bots like GPTBot and ClaudeBot?

What does the * wildcard mean in User-agent?

Can I set different rules for different bots?

Yes. Add separate rule blocks for each bot. For example, you might allow Googlebot to crawl everything while blocking Bingbot from /private/. Each bot follows its own most-specific matching block.

What is Crawl-delay and should I use it?