Skip to main content
SlapMyWeb
Get started
Skip to tool
SEO tools/All tools
SEO

Robots.txt Generator

Generate a properly formatted robots.txt file to control how search engine crawlers access your site.

Disallow
robots.txt
User-agent: *
Disallow:

What is Robots.txt Generator?

A robots.txt file is a plain text file placed at the root of your website that tells search engine crawlers which pages or sections they are allowed or disallowed from accessing. The Robots.txt Generator by SlapMyWeb lets you visually build this file without memorizing the syntax. You can add multiple user-agent blocks, set allow/disallow rules per path, configure crawl delays, and include your sitemap URL -- all with one-click presets for common configurations like blocking AI bots (GPTBot, ClaudeBot, Bytespider, CCBot) from scraping your content.

How to Use Robots.txt Generator

  1. 1

    Choose a preset or start from scratch

    Click "Allow All", "Block All", or "Block AI Bots" for instant configurations, or build custom rules manually.

  2. 2

    Configure user-agent rules

    Set the user-agent name (e.g., Googlebot, GPTBot, or * for all bots), then add Allow and Disallow paths for each agent.

  3. 3

    Add sitemap and crawl delay

    Enter your sitemap URL so crawlers discover all pages. Optionally set a crawl delay to protect your server from aggressive bots.

  4. 4

    Copy and upload

    Copy the generated robots.txt content and save it as a file named robots.txt in your website root directory.

Features

  • Visual editor with multiple user-agent blocks
  • One-click presets: Allow All, Block All, Block AI Bots
  • Block GPTBot, ChatGPT-User, ClaudeBot, Bytespider, CCBot
  • Add unlimited Allow and Disallow path rules
  • Sitemap URL inclusion for crawler discovery
  • Configurable crawl delay per user-agent
  • Real-time preview as you edit
  • One-click copy to clipboard

Related Tools

Meta Tag Generator

Create perfect SEO meta tags with OG + Twitter cards. Live character counters.

Open tool

Meta Tag Analyzer

Analyze any URL's meta tags, OG tags, Twitter cards and SEO elements with scoring.

Open tool

Open Graph Tag Generator

Generate Open Graph meta tags with live Facebook and LinkedIn preview.

Open tool

Open Graph Checker

Check any URL's Open Graph tags and preview how links appear on social media.

Open tool

Twitter Card Generator

Generate Twitter Card meta tags with live preview for summary and large image cards.

Open tool

Robots.txt Tester

Paste your robots.txt and test if a URL is crawlable by any user agent.

Open tool
See all SEO tools

Frequently Asked Questions

Where do I place the robots.txt file?+
The robots.txt file must be placed at the root of your domain, accessible at https://yourdomain.com/robots.txt. Search engine crawlers look for it at this exact location before crawling your site.
Does robots.txt block pages from appearing in Google?+
No. Robots.txt prevents crawling, not indexing. If other pages link to a disallowed URL, Google may still index it (showing a URL-only result). To prevent indexing, use a noindex meta tag or X-Robots-Tag HTTP header.
Should I block AI bots like GPTBot and ClaudeBot?+
That depends on your goals. If you don't want AI companies scraping your content for training data, blocking GPTBot, ChatGPT-User, ClaudeBot, Bytespider, and CCBot via robots.txt is the standard approach. However, this won't prevent content already scraped.
What does the * wildcard mean in User-agent?+
User-agent: * matches all crawlers that don't have a more specific rule. It's the default catch-all. Specific user-agent blocks (like Googlebot) take priority over the wildcard for that particular bot.
Can I set different rules for different bots?+
Yes. Add separate rule blocks for each bot. For example, you might allow Googlebot to crawl everything while blocking Bingbot from /private/. Each bot follows its own most-specific matching block.
What is Crawl-delay and should I use it?+
Crawl-delay tells bots to wait a specified number of seconds between requests. Google ignores this directive (use Search Console instead), but Bing, Yandex, and others respect it. Use it if aggressive crawling is straining your server.