Skip to content
Browse all guides

Issue #5 · critical

Issue 5

What is this issue?

A robots.txt file is a plain-text file placed at a website's root directory (e.g., https://example.com/robots.txt) that instructs web crawlers which parts of the site they can or cannot access. This issue checks whether a robots.txt file exists, is accessible, properly configured, and doesn't contain rules that block important pages from being crawled or indexed.

A passing implementation requires:

  • A robots.txt file accessible at the root domain returning 200 OK
  • Properly formatted directives following the Robots Exclusion Protocol
  • No rules that block important pages, CSS, or JavaScript files
  • No conflicting signals (e.g., blocking a URL that's in your sitemap)
  • Proper noindex directives via meta tags or HTTP headers (not in robots.txt)

Example: A properly configured robots.txt that allows all crawlers access to public pages while blocking access to private admin areas.

Why does it matter?

The robots.txt file is critical for SEO because it:

  • Controls Crawlability: Determines which pages search engines can crawl and which they must avoid
  • Protects Indexability: Prevents important pages from being blocked unintentionally
  • Manages Crawl Budget: Helps search engines focus on your most important content
  • Prevents Duplicate Content: Can block crawlers from accessing duplicate or low-value pages

Misconfigured robots.txt files can accidentally block your entire site from search engines or prevent important pages from being discovered. This directly impacts your SEO health score by affecting how search engines discover, crawl, and index your content.

How to fix it

  1. Ensure robots.txt exists: Create a robots.txt file at your root domain (/robots.txt) that returns 200 OK when accessed.

  2. Review Disallow rules: Check that you're not blocking important pages, CSS, or JavaScript files that search engines need to render your pages properly.

  3. Remove conflicting rules: Ensure URLs listed in your sitemap aren't blocked by robots.txt Disallow rules.

  4. Use proper noindex method: Don't use robots.txt to prevent indexing. Instead, use:

    • <meta name="robots" content="noindex"> in your HTML
    • X-Robots-Tag: noindex in your HTTP headers

  5. Declare your sitemap: Add a Sitemap: directive pointing to your XML sitemap location.

  6. Don't expose sensitive paths: Avoid listing private directories (like /admin/) in robots.txt, as it publicly reveals their existence.

  7. Validate regularly: Periodically check your robots.txt file to ensure it's accessible and properly configured.

Examples

Example 1: Missing robots.txt

Problematic State (Fails): A website has no robots.txt file at /robots.txt. Search engines don't know which pages they can or cannot crawl, which may lead to crawling of private or low-value pages.

Corrected State (Passes): Create a robots.txt file at https://example.com/robots.txt:

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

Example 2: Blocking Important Resources

Problematic State (Fails): A robots.txt file blocks CSS and JavaScript files:

User-agent: *
Disallow: /css/
Disallow: /js/

Search engines cannot render the page properly without these resources.

Corrected State (Passes): Allow access to CSS and JavaScript files:

User-agent: *
Allow: /
# Don't block CSS or JS files
Sitemap: https://example.com/sitemap.xml

Example 3: Conflicting Sitemap Rules

Problematic State (Fails): A sitemap contains a URL that's blocked by robots.txt:

# robots.txt
Disallow: /private-page

<!-- sitemap.xml -->
<url>
  <loc>https://example.com/private-page</loc>
</url>

Corrected State (Passes): Either remove the URL from the sitemap or remove the Disallow rule:

# robots.txt
# Allow crawling of private-page
Sitemap: https://example.com/sitemap.xml

How PixyScan detects this

PixyScan performs comprehensive robots.txt analysis through the following logical steps:

  1. File Discovery: PixyScan attempts to fetch the robots.txt file from your root domain and records the HTTP status code.

  2. Directive Parsing: If the file exists, PixyScan parses all directives line-by-line, including:

    • User-agent rules (which crawlers the rules apply to)
    • Disallow rules (what's blocked)
    • Allow rules (what's explicitly allowed)
    • Sitemap declarations
    • Crawl-delay settings

  3. Conflict Detection: PixyScan cross-references robots.txt rules against:

    • Your sitemap URLs (to find blocked sitemap entries)
    • Canonical URLs (to check if they're blocked)
    • CSS and JavaScript files (to ensure they're accessible)

  4. Noindex Detection: PixyScan checks for noindex directives in:

    • HTML <meta name="robots" content="noindex"> tags
    • HTTP X-Robots-Tag response headers

  5. Issue Identification: PixyScan raises issues when:

    • robots.txt is missing or returns an error (CRITICAL or WARNING)
    • Important pages or resources are blocked (CRITICAL)
    • Sitemap URLs conflict with Disallow rules (IMPORTANT)
    • Noindex directives are missing on private files (WARNING)
    • robots.txt hasn't been updated in over a year (SUGGESTION)

References