Issue 191

This issue reports that CCBot — the web crawler operated by Common Crawl, whose open corpus powers many open-source LLMs and AI research projects — is explicitly blocked in the site's robots.txt.

Having no CCBot rule at all is not a problem: a bot with no stanza of its own simply follows the wildcard (*) rules, which is how the vast majority of correctly configured sites work. The finding fires only when a User-agent: CCBot stanza exists and its rules deny the bot access to the site root.

The check passes when:

• robots.txt has no CCBot stanza and the wildcard rules allow crawling (the bot inherits *), or
• robots.txt has a CCBot stanza whose rules allow access to the site root

The issue fires when:

• A User-agent: CCBot stanza exists and its effective rules block the site root

Blocking CCBot can be a deliberate, legitimate business choice — the finding (severity STANDARD) exists to make sure that choice is visible and intentional, because the consequence is that the site will be absent from the Common Crawl corpus that many AI models train on.

Example (passes — no stanza, bot inherits *):

User-agent: *
Allow: /

Example (passes — explicitly allowed):

User-agent: CCBot
Allow: /

Example (fires — explicitly blocked):

User-agent: CCBot
Disallow: /

Open-Source AI Training: Common Crawl data is used to train many open-source Large Language Models (LLMs) like GPT-J, GPT-Neo, and others. Your content may contribute to these models if you allow CCBot.

Research and Archiving: Common Crawl provides web data for researchers, archivists, and non-commercial projects. Allowing CCBot supports open web data initiatives.

Content Control: If you don't want your content to be part of the Common Crawl dataset (and subsequently used in open-source AI training), you should explicitly disallow CCBot.

Public Dataset: Unlike some proprietary crawlers, Common Crawl data is publicly accessible, meaning your content could be used by anyone with access to the dataset.

Ethical Considerations: Some publishers prefer to opt-out of large-scale web scraping for AI training to protect their intellectual property or control how their content is used.

SEO Health Score: PixyScan raises this finding (severity STANDARD) only when CCBot is explicitly blocked in robots.txt. Having no CCBot rule at all is a pass — the bot simply inherits the wildcard (*) rules.

1. Decide your policy - Determine whether you want your content to be part of the Common Crawl dataset:

   • Allow: Your content will be part of the open web crawl dataset used for research and open-source AI
   • Disallow: Your content won't be included in Common Crawl's dataset

2. Add the rule to robots.txt - Add the following to your robots.txt file:

   User-agent: CCBot
   Disallow: /
   

   Or to allow:

   User-agent: CCBot
   Allow: /
   

3. Place it correctly - The rule can be placed anywhere in robots.txt, but many prefer to group AI bot rules together.

4. Test the rule - Verify the syntax is correct using Google Search Console's robots.txt tester or by checking the file directly.

5. Consider partial rules - You can disallow specific sections while allowing others:

   User-agent: CCBot
   Disallow: /private/
   Allow: /
   

Example 1: CCBot explicitly blocked (issue fires)

Problematic State (Fires): robots.txt contains a CCBot stanza that blocks the site root:

User-agent: CCBot
Disallow: /

Corrected State (Passes): If the block was unintentional, remove the stanza (CCBot then inherits the wildcard rules) or allow it explicitly:

User-agent: CCBot
Allow: /

If the block is a deliberate business choice, no change is needed — the STANDARD-severity finding simply records that the site will be absent from the Common Crawl corpus that many AI models train on.

Example 2: No CCBot rule at all (passes)

State (Passes):

User-agent: *
Allow: /

There is no CCBot stanza, so CCBot follows the wildcard (*) rules like any other crawler. This is a normal, correct configuration — no issue is raised.

Example 3: Partial restriction (passes)

State (Passes):

User-agent: CCBot
Disallow: /private/
Disallow: /admin/
Allow: /

The stanza restricts specific paths but the site root stays reachable — PixyScan evaluates the effective access to / with longest-match rules, so this passes.

PixyScan raises this issue only when CCBot is explicitly blocked in robots.txt — the absence of a CCBot stanza is a pass, because a bot with no stanza of its own simply inherits the wildcard (*) rules, which is a normal, correct configuration.

Detection steps (checkAiBotConfiguration in apps/crawler/robots-txt-parser.js, run once per scan by the GEO & AI Engine Signals toggle group):

1. Fetches robots.txt — the crawler requests /robots.txt from the domain being audited.

2. Parses all User-agent: groups — user-agent matching is case-insensitive (CCBot, ccbot, and CCBOT all match the same group).

3. Evaluates effective access to the site root (/) — using the same longest-match path logic real crawlers use (per RFC 9309): the most specific matching Allow/Disallow rule wins, and a bot-specific stanza overrides the wildcard * group. This is real rule evaluation, not a naive "does Disallow: / appear" string test.

4. Outcome:

   • No CCBot stanza, wildcard allows the root → pass (not-configured; the bot inherits *)
   • No CCBot stanza, wildcard blocks everything → pass here (the site-wide block is Issue #227's finding; it is not repeated per bot)
   • Stanza present, root allowed → pass (allowed)
   • Stanza present, root blocked → issue raised (blocked, severity STANDARD)

CCBot is a training bot: blocking it is a legitimate business choice, so the finding is reported at STANDARD severity to surface the consequence — the site will be absent from the Common Crawl corpus that many AI models train on — without treating it as a failure.