Skip to content
Browse all guides

Issue #187 · standard

Issue 187

What is this issue?

This issue reports that GPTBot — the web crawler used by OpenAI to collect data for training ChatGPT and other AI models — is explicitly blocked in the site's robots.txt.

Having no GPTBot rule at all is not a problem: a bot with no stanza of its own simply follows the wildcard (*) rules, which is how the vast majority of correctly configured sites work. The finding fires only when a User-agent: GPTBot stanza exists and its rules deny the bot access to the site root.

The check passes when:

  • robots.txt has no GPTBot stanza and the wildcard rules allow crawling (the bot inherits *), or
  • robots.txt has a GPTBot stanza whose rules allow access to the site root

The issue fires when:

  • A User-agent: GPTBot stanza exists and its effective rules block the site root

Blocking GPTBot can be a deliberate, legitimate business choice — the finding (severity STANDARD) exists to make sure that choice is visible and intentional, because the consequence is that the site's content will not be part of future GPT model knowledge.

Example (passes — no stanza, bot inherits *):

User-agent: *
Allow: /

Example (passes — explicitly allowed):

User-agent: GPTBot
Allow: /

Example (fires — explicitly blocked):

User-agent: GPTBot
Disallow: /

Why does it matter?

AI Training Data Control: GPTBot crawls websites to collect training data for OpenAI's language models. Having an explicit rule lets you control whether your content is used for AI training.

Content Protection: If you don't want your content to be used in AI training datasets, you should explicitly disallow GPTBot.

AI Visibility: Allowing GPTBot means your content may be referenced or cited in ChatGPT responses, potentially driving traffic to your site.

Legal and Ethical Considerations: Some publishers prefer to opt-out of AI training to protect their intellectual property or comply with content licensing policies.

SEO Health Score: PixyScan raises this finding (severity STANDARD) only when GPTBot is explicitly blocked in robots.txt. Having no GPTBot rule at all is a pass — the bot simply inherits the wildcard (*) rules.

How to fix it

  1. Decide your policy - Determine whether you want OpenAI to use your content for training:

    • Allow: Your content may appear in ChatGPT responses and training data
    • Disallow: Your content won't be used for ChatGPT training

  2. Add the rule to robots.txt - Add the following to your robots.txt file:

    User-agent: GPTBot
Disallow: /

    Or to allow:

    User-agent: GPTBot
Allow: /

  3. Place it correctly - The rule can be placed anywhere in robots.txt, but many prefer to group AI bot rules together.

  4. Test the rule - Verify the syntax is correct using Google Search Console's robots.txt tester or by checking the file directly.

  5. Consider partial rules - You can disallow specific sections while allowing others:

    User-agent: GPTBot
Disallow: /private/
Allow: /

Examples

Example 1: GPTBot explicitly blocked (issue fires)

Problematic State (Fires): robots.txt contains a GPTBot stanza that blocks the site root:

User-agent: GPTBot
Disallow: /

Corrected State (Passes): If the block was unintentional, remove the stanza (GPTBot then inherits the wildcard rules) or allow it explicitly:

User-agent: GPTBot
Allow: /

If the block is a deliberate business choice, no change is needed — the STANDARD-severity finding simply records that the site has opted out of future GPT model knowledge.

Example 2: No GPTBot rule at all (passes)

State (Passes):

User-agent: *
Allow: /

There is no GPTBot stanza, so GPTBot follows the wildcard (*) rules like any other crawler. This is a normal, correct configuration — no issue is raised.

Example 3: Partial restriction (passes)

State (Passes):

User-agent: GPTBot
Disallow: /private/
Disallow: /admin/
Allow: /

The stanza restricts specific paths but the site root stays reachable — PixyScan evaluates the effective access to / with longest-match rules, so this passes.

How PixyScan detects this

PixyScan raises this issue only when GPTBot is explicitly blocked in robots.txt — the absence of a GPTBot stanza is a pass, because a bot with no stanza of its own simply inherits the wildcard (*) rules, which is a normal, correct configuration.

Detection steps (checkAiBotConfiguration in apps/crawler/robots-txt-parser.js, run once per scan by the GEO & AI Engine Signals toggle group):

  1. Fetches robots.txt — the crawler requests /robots.txt from the domain being audited.

  2. Parses all User-agent: groups — user-agent matching is case-insensitive (GPTBot, gptbot, and GPTBOT all match the same group).

  3. Evaluates effective access to the site root (/) — using the same longest-match path logic real crawlers use (per RFC 9309): the most specific matching Allow/Disallow rule wins, and a bot-specific stanza overrides the wildcard * group. This is real rule evaluation, not a naive "does Disallow: / appear" string test.

  4. Outcome:

    • No GPTBot stanza, wildcard allows the root → pass (not-configured; the bot inherits *)
    • No GPTBot stanza, wildcard blocks everything → pass here (the site-wide block is Issue #227's finding; it is not repeated per bot)
    • Stanza present, root allowed → pass (allowed)
    • Stanza present, root blocked → issue raised (blocked, severity STANDARD)

GPTBot is a training bot: blocking it is a legitimate business choice, so the finding is reported at STANDARD severity to surface the consequence — OpenAI's crawler cannot read the site, so its content will not be part of future GPT model knowledge — without treating it as a failure.

References