What is the difference between a training crawler and an answer crawler?

A training crawler (GPTBot, ClaudeBot, CCBot, Google-Extended) collects text that may be used to train a future model. An answer crawler (ChatGPT-User, OAI-SearchBot, Claude-Web, Perplexity-User) fetches your page at the moment a user asks a question, so it can cite you in the reply. Blocking the first is an opt-out from training. Blocking the second removes you from answers being generated today.

My robots.txt does not mention any AI crawler. Is that bad?

No — it means they are all allowed. robots.txt is an opt-out mechanism: anything not disallowed is permitted. If you want to stay out of model training you have to say so explicitly.

Does blocking GPTBot remove my content from ChatGPT?

It stops OpenAI using your pages for training going forward. It does not remove anything already learned, and it does not affect ChatGPT-User, which is the agent that fetches your page live when someone asks about you. Those are separate user-agents and need separate rules.

Does robots.txt actually stop these crawlers?

It is a request, not an enforcement mechanism. The major crawlers listed here document that they honour it. A crawler that chooses to ignore robots.txt can only be stopped at the server or firewall.

What is Google-Extended?

It is not a crawler. It is a token you can disallow to opt out of your content being used for Google's generative AI training, while leaving normal Googlebot indexing untouched. Disallowing Google-Extended does not affect your Search rankings.

Why does a crawler say "via *"?

Because there is no rule naming that crawler, so the wildcard group applies to it. The verdict is real, but it comes from your catch-all rule rather than a deliberate decision about that agent.

SEO

AI Crawler Checker

See which AI crawlers your robots.txt lets in — and whether you are blocking the ones that answer questions about you today.

100% free
No sign-up

Your robots.txt

What this tool does

Checks all 14 AI crawlers in one pass
Separates training crawlers from live-answer crawlers
Reads your live robots.txt, or paste your own
Explains what each block actually costs you
Free, no account, no daily limit

Want the whole picture? Run a free website audit for a score and prioritised issues. No signup needed to start.Run a free audit

The short version

What is AI Crawler Checker?

AI Crawler Checker reads a site's robots.txt and reports, crawler by crawler, which AI systems are allowed to read it.

It checks fourteen agents — GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot, Perplexity-User, CCBot, Google-Extended, Bytespider, Applebot-Extended, FacebookBot and Meta-ExternalAgent — and applies the same precedence rules a real crawler does: an exact user-agent group beats the wildcard group, the longest matching path wins, and consecutive User-agent lines share one set of rules. The distinction the report is built around is training versus live answers. Blocking a training crawler keeps your text out of the next model. Blocking a live-answer crawler makes you invisible in the assistant somebody is using right now, which is usually not what a site intends when it copies a robots.txt snippet from a blog post.

Step by step

How to use AI Crawler Checker

1
Enter your domain
Type your site and press Fetch robots.txt. The file is read from your live site, exactly as a crawler would read it.
2
Or paste a draft
Paste robots.txt straight into the box to test a change before you ship it.
3
Read the verdict row by row
Each crawler shows Allowed, Blocked, or Not mentioned. "Not mentioned" means allowed by default — robots.txt is an opt-out, not an opt-in.
4
Check the purpose column
Sort your intent out here. Blocking "Model training" is a deliberate, common choice. Blocking "Live answers" costs you visibility today.

What's inside

Everything this tool
does for you

All 14 named AI crawlers checked at once
Correct robots.txt precedence: exact agent over wildcard, longest path wins
Handles grouped User-agent lines, which naive parsers get wrong
Training vs live-answer intent for every crawler
Reads your live file or a pasted draft
Free, no signup, no daily limit

Understand the result

Related free tools

All SEO tools

Meta Tag Generator

Import metadata from a URL or edit SEO and social tags, preview them, and copy the generated HTML.

SEO

Meta Tag Analyzer

Inspect page metadata, social tags, response directives and actionable fixes from a URL.

SEO

Open Graph Tag Generator

Generate Open Graph meta tags with live Facebook and LinkedIn preview.

SEO

Open Graph Checker

Check any URL's Open Graph tags and preview how links appear on social media.

SEO

Twitter Card Generator

Generate Twitter Card meta tags with live preview for summary and large image cards.

SEO

Robots.txt Generator

Build a robots.txt file with a visual editor. Add rules for any user agent.

SEO

Questions

Frequently asked questions

Beyond this tool

See the full picture for your website.

Enter your URL for a free audit across eight scored areas. See prioritised issues and clear next steps in one report.

Run a free website audit

Free to start · No signup required