Browse all guides

Issue #189 · critical

Issue 189

What is this issue?

This issue reports that PerplexityBot — the crawler Perplexity AI uses to fetch and index pages so they can be cited in its answers — is explicitly blocked in the site's robots.txt.

Having no PerplexityBot rule at all is not a problem: a bot with no stanza of its own simply follows the wildcard (*) rules, which is how the vast majority of correctly configured sites work. The finding fires only when a User-agent: PerplexityBot stanza exists and its rules deny the bot access to the site root.

The check passes when:

robots.txt has no PerplexityBot stanza and the wildcard rules allow crawling (the bot inherits *), or
robots.txt has a PerplexityBot stanza whose rules allow access to the site root

The issue fires when:

A User-agent: PerplexityBot stanza exists and its effective rules block the site root

PerplexityBot is a retrieval bot, not a training bot: it fetches pages to answer live user queries. Blocking it removes the site from Perplexity's answers entirely — the site cannot be fetched, indexed, or cited. Because that is a direct visibility loss, this finding is severity CRITICAL (the four training-bot findings, #187/#188/#190/#191, are STANDARD).

Example (passes — no stanza, bot inherits *):

User-agent: *
Allow: /

Example (passes — explicitly allowed):

User-agent: PerplexityBot
Allow: /

Example (fires — explicitly blocked):

User-agent: PerplexityBot
Disallow: /

Why does it matter?

AI Search Visibility: Perplexity AI is an emerging AI search engine that answers questions by citing web sources. Allowing PerplexityBot means your content may be referenced in Perplexity answers, driving referral traffic.

Content Attribution: Perplexity AI typically provides citations and links back to source content, offering a potential new traffic source different from traditional search engines.

Early Adoption Advantage: As AI search engines gain market share, being indexed early may provide competitive advantages.

Content Protection: If you don't want your content to be used in Perplexity AI responses, you should explicitly disallow PerplexityBot.

Control Over AI Training: Unlike some AI bots that use data for model training, PerplexityBot primarily uses crawled content for real-time answer generation with attribution.

SEO Health Score: PixyScan raises this finding (severity CRITICAL) only when PerplexityBot is explicitly blocked in robots.txt — blocking a retrieval bot removes the site from Perplexity answers. Having no PerplexityBot rule at all is a pass — the bot simply inherits the wildcard (*) rules.

How to fix it

Decide your policy - Determine whether you want Perplexity AI to crawl and cite your content:
- Allow: Your content may be cited in Perplexity AI answers with attribution
- Disallow: Your content won't be crawled or cited by Perplexity AI
Add the rule to robots.txt - Add the following to your robots.txt file:
```
User-agent: PerplexityBot
Disallow: /
```
Or to allow:
```
User-agent: PerplexityBot
Allow: /
```
Place it correctly - The rule can be placed anywhere in robots.txt, but many prefer to group AI bot rules together.
Test the rule - Verify the syntax is correct using Google Search Console's robots.txt tester or by checking the file directly.
Consider partial rules - You can disallow specific sections while allowing others:
```
User-agent: PerplexityBot
Disallow: /private/
Allow: /
```

Examples

Example 1: PerplexityBot explicitly blocked (issue fires — CRITICAL)

Problematic State (Fires): robots.txt contains a PerplexityBot stanza that blocks the site root:

User-agent: PerplexityBot
Disallow: /

PerplexityBot is a retrieval bot — this block means Perplexity cannot fetch or index the pages, so the site cannot be cited in Perplexity answers at all.

Corrected State (Passes): Remove the stanza (PerplexityBot then inherits the wildcard rules) or allow it explicitly:

User-agent: PerplexityBot
Allow: /

Example 2: No PerplexityBot rule at all (passes)

State (Passes):

User-agent: *
Allow: /

There is no PerplexityBot stanza, so PerplexityBot follows the wildcard (*) rules like any other crawler. This is a normal, correct configuration — no issue is raised.

Example 3: Partial restriction (passes)

State (Passes):

User-agent: PerplexityBot
Disallow: /private/
Disallow: /admin/
Allow: /

The stanza restricts specific paths but the site root stays reachable — PixyScan evaluates the effective access to / with longest-match rules, so this passes.

How PixyScan detects this

PixyScan raises this issue only when PerplexityBot is explicitly blocked in robots.txt — the absence of a PerplexityBot stanza is a pass, because a bot with no stanza of its own simply inherits the wildcard (*) rules, which is a normal, correct configuration.

Detection steps (checkAiBotConfiguration in apps/crawler/robots-txt-parser.js, run once per scan by the GEO & AI Engine Signals toggle group):

Fetches robots.txt — the crawler requests /robots.txt from the domain being audited.
Parses all User-agent: groups — user-agent matching is case-insensitive (PerplexityBot, perplexitybot, and PERPLEXITYBOT all match the same group).
Evaluates effective access to the site root (/) — using the same longest-match path logic real crawlers use (per RFC 9309): the most specific matching Allow/Disallow rule wins, and a bot-specific stanza overrides the wildcard * group. This is real rule evaluation, not a naive "does Disallow: / appear" string test.
Outcome:
- No PerplexityBot stanza, wildcard allows the root → pass (not-configured; the bot inherits *)
- No PerplexityBot stanza, wildcard blocks everything → pass here (the site-wide block is Issue #227's finding; it is not repeated per bot)
- Stanza present, root allowed → pass (allowed)
- Stanza present, root blocked → issue raised (blocked, severity CRITICAL)

Unlike the AI training bots (#187, #188, #190, #191), PerplexityBot is a retrieval bot: it fetches pages to answer live user queries. Blocking it removes the site from Perplexity's answers entirely — Perplexity cannot fetch or index the pages, so the site cannot be cited. That direct visibility loss is why this finding is CRITICAL while the training-bot findings are STANDARD.