Skip to content
Browse all guides

Issue #175 · critical

Issue 175

What is this issue?

This issue checks whether the canonical URL specified on a page is accessible to search engine crawlers and not blocked by the robots.txt file. If a canonical page is blocked, search engines cannot validate and honor the canonical directive.

A passing implementation means:

  • The canonical URL is NOT disallowed in robots.txt
  • The canonical URL returns a 200 OK status (not blocked, not redirected)
  • Search engines can crawl the canonical URL to validate the canonical relationship

Example:

  • Page URL: https://example.com/product?color=blue
  • Canonical URL: https://example.com/product
  • robots.txt: Disallow: /product ❌ This would block the canonical!
  • Correct: No disallow for /product ✅

Why does it matter?

Canonical Validation Failure: Google explicitly states that if the canonical URL is blocked by robots.txt, they cannot crawl it and therefore cannot validate the canonical relationship. The canonical directive may be ignored.

Wrong Canonical Selection: Without access to the canonical URL, search engines may choose a different canonical than intended, potentially indexing the wrong version of the page.

Indexing Issues: Pages with blocked canonicals may not be indexed properly, or may be indexed with the wrong URL.

Wasted Canonical Tags: If the canonical URL is blocked, the canonical tag is effectively useless and provides no SEO benefit.

SEO Health Score: Resolving this issue is critical for proper canonical implementation and significantly improves the technical SEO score.

How to fix it

  1. Identify canonical URLs - Review your pages to determine what canonical URLs are specified in the <link rel="canonical"> tags.

  2. Check robots.txt - Examine your robots.txt file to see if any canonical URLs or their parent paths are disallowed.

  3. Remove blocking rules - If a canonical URL is blocked, either:

    • Remove the Disallow directive that blocks the canonical URL, OR
    • Change the canonical URL to a non-blocked URL, OR
    • Use meta robots with noindex instead of robots.txt blocking for the non-canonical versions

  4. Verify accessibility - Ensure the canonical URL:

    • Returns HTTP 200 status
    • Is not blocked by robots.txt
    • Is not noindexed
    • Is accessible to crawlers

  5. Test with Search Console - Use the URL Inspection tool in Google Search Console to verify Google can crawl the canonical URL.

Examples

Example 1: Basic Canonical Blocked

Problematic State (Fails):

  • Page: https://example.com/product?color=blue
  • Canonical: https://example.com/product
  • robots.txt: Disallow: /product

The canonical URL is blocked, so search engines cannot validate it.

Corrected State (Passes):

  • Remove the Disallow rule for the canonical URL:
User-agent: *
Disallow: /admin/
# Don't block /product

Example 2: Partial Path Blocked

Problematic State (Fails):

  • Page: https://example.com/blog/post-1
  • Canonical: https://example.com/blog/post-1
  • robots.txt: Disallow: /blog

The entire blog section is blocked, including the canonical URL.

Corrected State (Passes):

  • Allow the canonical URL path:
User-agent: *
Allow: /blog/post-1
Disallow: /blog/admin/

Example 3: Using Meta Robots Instead

Problematic State (Fails):

  • Non-canonical URLs are blocked by robots.txt
  • But canonical URL is also blocked

Corrected State (Passes):

  • Use meta robots for non-canonical versions:
<!-- On non-canonical URL -->
<meta name="robots" content="noindex" />
  • Don't block the canonical URL in robots.txt

How PixyScan detects this

PixyScan performs the following checks:

  1. Extracts canonical URL - For each page crawled, PixyScan extracts the canonical URL from the <link rel="canonical"> tag.

  2. Fetches robots.txt - The system retrieves and parses the site's robots.txt file.

  3. Checks if canonical is blocked - PixyScan tests whether the canonical URL matches any Disallow pattern in robots.txt.

  4. Validates accessibility - The system also checks if the canonical URL:

    • Returns a 200 OK status
    • Is not redirected
    • Is not noindexed

  5. Reports issues - If the canonical URL is blocked by robots.txt or inaccessible, the issue is flagged with details about which rule is blocking it.

References