Issue 190

This issue reports that Google-Extended — the robots.txt control token (separate from Googlebot) that governs whether content is used for Google's generative AI products such as Gemini — is explicitly blocked in the site's robots.txt.

Having no Google-Extended rule at all is not a problem: a token with no stanza of its own simply follows the wildcard (*) rules, which is how the vast majority of correctly configured sites work. The finding fires only when a User-agent: Google-Extended stanza exists and its rules deny access to the site root.

The check passes when:

• robots.txt has no Google-Extended stanza and the wildcard rules allow crawling (the token inherits *), or
• robots.txt has a Google-Extended stanza whose rules allow access to the site root

The issue fires when:

• A User-agent: Google-Extended stanza exists and its effective rules block the site root

Blocking Google-Extended can be a deliberate, legitimate business choice — the finding (severity STANDARD) exists to make sure that choice is visible and intentional, because the consequence is that Google will not use the site content for Gemini training or grounding. It does not affect regular Googlebot search indexing.

Example (passes — no stanza, token inherits *):

User-agent: *
Allow: /

Example (passes — explicitly allowed):

User-agent: Google-Extended
Allow: /

Example (fires — explicitly blocked):

User-agent: Google-Extended
Disallow: /

Separate from Googlebot: Google-Extended is a distinct crawler from regular Googlebot. Blocking or allowing Googlebot does NOT affect Google-Extended, and vice versa.

Generative AI Training: Google-Extended is used to crawl content for Google's generative AI products (Bard/Gemini). This is content used for AI model training, not regular search indexing.

Content Control: Having an explicit rule lets you control whether your content contributes to Google's AI training datasets while still allowing regular Google search indexing.

Strategic Decision: Publishers must decide whether to contribute content to AI training:

• Allow: Support AI development, potential citation in AI responses
• Disallow: Protect content from being used in AI training datasets

Emerging Standard: As Google expands its generative AI offerings, controlling Google-Extended is becoming an important part of robots.txt management.

SEO Health Score: PixyScan raises this finding (severity STANDARD) only when Google-Extended is explicitly blocked in robots.txt. Having no Google-Extended rule at all is a pass — the token simply inherits the wildcard (*) rules.

1. Decide your policy - Determine whether you want Google to use your content for generative AI training:

   • Allow: Your content may be used in Google's AI products (Bard/Gemini)
   • Disallow: Your content won't be used for Google's AI training

2. Note the distinction - Remember:

   • Googlebot = regular search indexing (usually want to allow)
   • Google-Extended = generative AI training (separate decision)

3. Add the rule to robots.txt - Add the following to your robots.txt file:

   User-agent: Google-Extended
   Disallow: /
   

   Or to allow:

   User-agent: Google-Extended
   Allow: /
   

4. Test the rule - Verify the syntax is correct using Google Search Console's robots.txt tester or by checking the file directly.

5. Consider partial rules - You can disallow specific sections while allowing others:

   User-agent: Google-Extended
   Disallow: /private/
   Allow: /
   

Example 1: Google-Extended explicitly blocked (issue fires)

Problematic State (Fires): robots.txt contains a Google-Extended stanza that blocks the site root:

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Disallow: /

Corrected State (Passes): If the block was unintentional, remove the stanza (Google-Extended then inherits the wildcard rules) or allow it explicitly:

User-agent: Google-Extended
Allow: /

If the block is a deliberate business choice, no change is needed — the STANDARD-severity finding simply records that Google will not use the content for Gemini training or grounding. Regular Googlebot search indexing is unaffected either way.

Example 2: No Google-Extended rule at all (passes)

State (Passes):

User-agent: *
Allow: /

There is no Google-Extended stanza, so the token follows the wildcard (*) rules. This is a normal, correct configuration — no issue is raised.

Example 3: Partial restriction (passes)

State (Passes):

User-agent: Google-Extended
Disallow: /private/
Disallow: /admin/
Allow: /

The stanza restricts specific paths but the site root stays reachable — PixyScan evaluates the effective access to / with longest-match rules, so this passes.

PixyScan raises this issue only when Google-Extended is explicitly blocked in robots.txt — the absence of a Google-Extended stanza is a pass, because a token with no stanza of its own simply inherits the wildcard (*) rules, which is a normal, correct configuration.

Detection steps (checkAiBotConfiguration in apps/crawler/robots-txt-parser.js, run once per scan by the GEO & AI Engine Signals toggle group):

1. Fetches robots.txt — the crawler requests /robots.txt from the domain being audited.

2. Parses all User-agent: groups — user-agent matching is case-insensitive (Google-Extended, google-extended, and GOOGLE-EXTENDED all match the same group).

3. Evaluates effective access to the site root (/) — using the same longest-match path logic real crawlers use (per RFC 9309): the most specific matching Allow/Disallow rule wins, and a token-specific stanza overrides the wildcard * group. This is real rule evaluation, not a naive "does Disallow: / appear" string test.

4. Outcome:

   • No Google-Extended stanza, wildcard allows the root → pass (not-configured; the token inherits *)
   • No Google-Extended stanza, wildcard blocks everything → pass here (the site-wide block is Issue #227's finding; it is not repeated per bot)
   • Stanza present, root allowed → pass (allowed)
   • Stanza present, root blocked → issue raised (blocked, severity STANDARD)

Google-Extended is a training control token: blocking it is a legitimate business choice, so the finding is reported at STANDARD severity to surface the consequence — Google will not use the site content for Gemini training or grounding — without treating it as a failure. Blocking Google-Extended does not affect regular Googlebot search indexing.