Browse all guides

Issue #188 · standard

Issue 188

What is this issue?

This issue reports that ClaudeBot — the web crawler used by Anthropic to collect data for training Claude AI models — is explicitly blocked in the site's robots.txt.

Having no ClaudeBot rule at all is not a problem: a bot with no stanza of its own simply follows the wildcard (*) rules, which is how the vast majority of correctly configured sites work. The finding fires only when a User-agent: ClaudeBot stanza exists and its rules deny the bot access to the site root.

The check passes when:

robots.txt has no ClaudeBot stanza and the wildcard rules allow crawling (the bot inherits *), or
robots.txt has a ClaudeBot stanza whose rules allow access to the site root

The issue fires when:

A User-agent: ClaudeBot stanza exists and its effective rules block the site root

Blocking ClaudeBot can be a deliberate, legitimate business choice — the finding (severity STANDARD) exists to make sure that choice is visible and intentional, because the consequence is that the site's content will not be part of future Claude model knowledge.

Example (passes — no stanza, bot inherits *):

User-agent: *
Allow: /

Example (passes — explicitly allowed):

User-agent: ClaudeBot
Allow: /

Example (fires — explicitly blocked):

User-agent: ClaudeBot
Disallow: /

Why does it matter?

AI Training Data Control: ClaudeBot crawls websites to collect training data for Anthropic's Claude AI models. Having an explicit rule lets you control whether your content is used for AI training.

Content Protection: If you don't want your content to be used in Anthropic's AI training datasets, you should explicitly disallow ClaudeBot.

AI Visibility: Allowing ClaudeBot means your content may be referenced or cited in Claude responses, potentially driving traffic to your site.

Legal and Ethical Considerations: Some publishers prefer to opt-out of AI training to protect their intellectual property or comply with content licensing policies.

Emerging Standard: As AI crawlers become more prevalent, having explicit rules demonstrates technical sophistication and awareness of the evolving search landscape.

SEO Health Score: PixyScan raises this finding (severity STANDARD) only when ClaudeBot is explicitly blocked in robots.txt. Having no ClaudeBot rule at all is a pass — the bot simply inherits the wildcard (*) rules.

How to fix it

Decide your policy - Determine whether you want Anthropic to use your content for training Claude:
- Allow: Your content may appear in Claude responses and training data
- Disallow: Your content won't be used for Claude training
Add the rule to robots.txt - Add the following to your robots.txt file:
```
User-agent: ClaudeBot
Disallow: /
```
Or to allow:
```
User-agent: ClaudeBot
Allow: /
```
Place it correctly - The rule can be placed anywhere in robots.txt, but many prefer to group AI bot rules together.
Test the rule - Verify the syntax is correct using Google Search Console's robots.txt tester or by checking the file directly.
Consider partial rules - You can disallow specific sections while allowing others:
```
User-agent: ClaudeBot
Disallow: /private/
Allow: /
```

Examples

Example 1: ClaudeBot explicitly blocked (issue fires)

Problematic State (Fires): robots.txt contains a ClaudeBot stanza that blocks the site root:

User-agent: ClaudeBot
Disallow: /

Corrected State (Passes): If the block was unintentional, remove the stanza (ClaudeBot then inherits the wildcard rules) or allow it explicitly:

User-agent: ClaudeBot
Allow: /

If the block is a deliberate business choice, no change is needed — the STANDARD-severity finding simply records that the site has opted out of future Claude model knowledge.

Example 2: No ClaudeBot rule at all (passes)

State (Passes):

User-agent: *
Allow: /

There is no ClaudeBot stanza, so ClaudeBot follows the wildcard (*) rules like any other crawler. This is a normal, correct configuration — no issue is raised.

Example 3: Partial restriction (passes)

State (Passes):

User-agent: ClaudeBot
Disallow: /private/
Disallow: /admin/
Allow: /

The stanza restricts specific paths but the site root stays reachable — PixyScan evaluates the effective access to / with longest-match rules, so this passes.

How PixyScan detects this

PixyScan raises this issue only when ClaudeBot is explicitly blocked in robots.txt — the absence of a ClaudeBot stanza is a pass, because a bot with no stanza of its own simply inherits the wildcard (*) rules, which is a normal, correct configuration.

Detection steps (checkAiBotConfiguration in apps/crawler/robots-txt-parser.js, run once per scan by the GEO & AI Engine Signals toggle group):

Fetches robots.txt — the crawler requests /robots.txt from the domain being audited.
Parses all User-agent: groups — user-agent matching is case-insensitive (ClaudeBot, claudebot, and CLAUDEBOT all match the same group).
Evaluates effective access to the site root (/) — using the same longest-match path logic real crawlers use (per RFC 9309): the most specific matching Allow/Disallow rule wins, and a bot-specific stanza overrides the wildcard * group. This is real rule evaluation, not a naive "does Disallow: / appear" string test.
Outcome:
- No ClaudeBot stanza, wildcard allows the root → pass (not-configured; the bot inherits *)
- No ClaudeBot stanza, wildcard blocks everything → pass here (the site-wide block is Issue #227's finding; it is not repeated per bot)
- Stanza present, root allowed → pass (allowed)
- Stanza present, root blocked → issue raised (blocked, severity STANDARD)

ClaudeBot is a training bot: blocking it is a legitimate business choice, so the finding is reported at STANDARD severity to surface the consequence — Anthropic's crawler cannot read the site, so its content will not be part of future Claude model knowledge — without treating it as a failure.