Where do I place the robots.txt file?

The robots.txt file must be placed at the root of your domain, accessible at https://yourdomain.com/robots.txt. Search engine crawlers look for it at this exact location before crawling your site.

Does robots.txt block pages from appearing in Google?

No. Robots.txt prevents crawling, not indexing. If other pages link to a disallowed URL, Google may still index it (showing a URL-only result). To prevent indexing, use a noindex meta tag or X-Robots-Tag HTTP header.

Should I block AI bots like GPTBot and ClaudeBot?

That depends on your goals. If you don't want AI companies scraping your content for training data, blocking GPTBot, ChatGPT-User, ClaudeBot, Bytespider, and CCBot via robots.txt is the standard approach. However, this won't prevent content already scraped.

What does the * wildcard mean in User-agent?

User-agent: * matches all crawlers that don't have a more specific rule. It's the default catch-all. Specific user-agent blocks (like Googlebot) take priority over the wildcard for that particular bot.

Can I set different rules for different bots?

Yes. Add separate rule blocks for each bot. For example, you might allow Googlebot to crawl everything while blocking Bingbot from /private/. Each bot follows its own most-specific matching block.

What is Crawl-delay and should I use it?

Crawl-delay tells bots to wait a specified number of seconds between requests. Google ignores this directive (use Search Console instead), but Bing, Yandex, and others respect it. Use it if aggressive crawling is straining your server.

SEO

Robots.txt Generator

Generate a properly formatted robots.txt file to control how search engine crawlers access your site.

100% free
No sign-up
No limits
Instant results

Quick Presets

User-Agent

Disallow

Crawl Delay (seconds, optional)

Sitemap URL

robots.txt

User-agent: *
Disallow:

What this tool does

Control which bots can crawl your site
Block AI bots from scraping your content
Set crawl delays to protect server resources
Include sitemap URL for better discovery
One-click presets for common configurations

Want the whole picture? Run 240+ checks on your site — free, 30 seconds, no signup.Run a free audit

The short version

What is Robots.txt Generator?

A robots.txt file is a plain text file placed at the root of your website that tells search engine crawlers which pages or sections they are allowed or disallowed from accessing.

The Robots.txt Generator by SlapMyWeb lets you visually build this file without memorizing the syntax. You can add multiple user-agent blocks, set allow/disallow rules per path, configure crawl delays, and include your sitemap URL -- all with one-click presets for common configurations like blocking AI bots (GPTBot, ClaudeBot, Bytespider, CCBot) from scraping your content.

Step by step

How to use Robots.txt Generator

1
Choose a preset or start from scratch
Click "Allow All", "Block All", or "Block AI Bots" for instant configurations, or build custom rules manually.
2
Configure user-agent rules
Set the user-agent name (e.g., Googlebot, GPTBot, or * for all bots), then add Allow and Disallow paths for each agent.
3
Add sitemap and crawl delay
Enter your sitemap URL so crawlers discover all pages. Optionally set a crawl delay to protect your server from aggressive bots.
4
Copy and upload
Copy the generated robots.txt content and save it as a file named robots.txt in your website root directory.

What's inside

Everything this tool
does for you

Visual editor with multiple user-agent blocks
One-click presets: Allow All, Block All, Block AI Bots
Block GPTBot, ChatGPT-User, ClaudeBot, Bytespider, CCBot
Add unlimited Allow and Disallow path rules
Sitemap URL inclusion for crawler discovery
Configurable crawl delay per user-agent
Real-time preview as you edit
One-click copy to clipboard

Understand the result

Related free tools

All SEO tools

Meta Tag Generator

Create perfect SEO meta tags with OG + Twitter cards. Live character counters.

SEO

Meta Tag Analyzer

Analyze any URL's meta tags, OG tags, Twitter cards and SEO elements with scoring.

SEO

Open Graph Tag Generator

Generate Open Graph meta tags with live Facebook and LinkedIn preview.

SEO

Open Graph Checker

Check any URL's Open Graph tags and preview how links appear on social media.

SEO

Twitter Card Generator

Generate Twitter Card meta tags with live preview for summary and large image cards.

SEO

Robots.txt Tester

Paste your robots.txt and test if a URL is crawlable by any user agent.

SEO

Questions

Frequently asked questions

This tool fixed one thing.
Now fix the whole site.

Our scanner runs 240+ checks across 10 audit areas, ranks every issue by the score points it costs, and hands you the code to paste.

Run a free audit

240+ checksResults in 30sNo signup