Skip to content
Browse all guides

Issue #227 · important

Issue 227

What is this issue?

This issue performs an aggregate indexability check on critical pages to confirm they are indexable by search engines. It combines multiple factors: noindex meta tags, robots.txt disallow rules, and HTTP status codes to determine if important pages are properly accessible to search engines.

A passing implementation means:

  • Critical pages (homepage, main category pages, key product/service pages) return HTTP 200 status
  • These pages do NOT have noindex meta tags
  • These pages are NOT disallowed in robots.txt
  • These pages are accessible and indexable by search engines

Example of indexable page:

  • URL: https://example.com/important-page
  • HTTP Status: 200 OK
  • Meta robots: <meta name="robots" content="index, follow"> (or not present)
  • robots.txt: Not disallowed ✅

Example of non-indexable page:

  • URL: https://example.com/important-page
  • HTTP Status: 200 OK
  • Meta robots: <meta name="robots" content="noindex"> ❌
  • Or robots.txt: Disallow: /important-page ❌

Why does it matter?

Search Visibility: If critical pages are not indexable, they won't appear in search results, resulting in zero organic traffic from those pages.

Business Impact: The most important pages on your site (homepage, main product pages, key conversion pages) must be indexable to drive business results.

Combined Barriers: A page can be blocked from indexing in multiple ways simultaneously (noindex + robots.txt disallow + 404 status). This check identifies all barriers in one audit.

Crawl Budget Optimization: Ensuring critical pages are indexable while non-critical pages are not helps search engines focus their crawl budget on valuable content.

SEO Health Score: This aggregate check is a fundamental validation that significantly impacts the overall technical SEO health score.

How to fix it

  1. Identify critical pages - Determine which pages are essential for SEO and business goals:

    • Homepage
    • Main category/section pages
    • Key product/service pages
    • Conversion pages (contact, pricing, etc.)

  2. Check each barrier - For each critical page, verify:

    a. HTTP Status: Page should return 200 OK (not 404, 500, etc.)

    b. Meta Robots: Page should NOT have noindex:

    <!-- Remove this if present on critical pages -->
<meta name="robots" content="noindex" />

    c. Robots.txt: Page should NOT be disallowed:

    # Remove this rule if it blocks critical pages
Disallow: /important-page

  3. Fix issues - Depending on what's blocking indexability:

    • Remove noindex meta tags from critical pages
    • Remove Disallow rules that block critical pages
    • Fix server errors (500) or missing pages (404)

  4. Validate with Search Console - Use the URL Inspection tool to verify Google sees the page as indexable.

  5. Monitor regularly - Set up alerts or regular audits to ensure critical pages remain indexable.

Examples

Example 1: Critical Page with noindex

Problematic State (Fails): Homepage has noindex tag:

<head>
  <meta name="robots" content="noindex" />
</head>

This prevents the homepage from being indexed.

Corrected State (Passes): Remove noindex tag from critical pages:

<head>
  <!-- No robots meta tag, or explicitly set to index -->
  <meta name="robots" content="index, follow" />
</head>

Example 2: Critical Page Blocked by robots.txt

Problematic State (Fails): Main product page is blocked:

# robots.txt
Disallow: /products/main-product

Corrected State (Passes): Remove the disallow rule for critical pages:

# robots.txt
# Don't block critical product pages
Allow: /products/main-product

Example 3: Multiple Barriers

Problematic State (Fails): Critical page has multiple indexability barriers:

  • Returns 200 OK
  • Has <meta name="robots" content="noindex">
  • Is disallowed in robots.txt

Corrected State (Passes): Remove all barriers:

  • Keep 200 OK status
  • Remove noindex meta tag
  • Remove Disallow rule from robots.txt

How PixyScan detects this

PixyScan performs the following aggregate checks:

  1. Identifies critical pages - The system identifies important pages based on:

    • Homepage (always checked)
    • Pages with high internal link counts
    • Pages specified in sitemaps
    • Pages with high traffic or conversion value

  2. Checks HTTP status - For each critical page, PixyScan verifies the HTTP status code is 200 OK.

  3. Checks meta robots - PixyScan parses the HTML and looks for noindex in:

    • <meta name="robots" content="noindex"> tags
    • X-Robots-Tag: noindex HTTP headers

  4. Checks robots.txt - PixyScan tests if the page URL matches any Disallow pattern in robots.txt.

  5. Aggregates results - PixyScan combines all three checks and flags pages that are blocked by any barrier.

  6. Reports issues - The issue is flagged if any critical page is:

    • Returning non-200 status (CRITICAL)
    • Having noindex tag (CRITICAL)
    • Blocked by robots.txt (CRITICAL)

References