Skip to content
Browse all guides

Issue #3 · critical

Issue 3

What is this issue?

A sitemap is an XML file that lists a website's URLs and metadata to guide search engine crawlers. This issue checks whether a sitemap exists, is accessible, properly formatted, and contains valid URLs that align with SEO best practices.

A passing implementation requires:

  • A sitemap file accessible at the root domain or declared in robots.txt
  • Valid XML format following sitemap protocol standards
  • URLs that return 200 OK status codes
  • No conflicting signals (e.g., URLs blocked by robots.txt or marked with noindex)
  • Proper canonical URL alignment with <loc> entries

Example: A properly configured sitemap at https://example.com/sitemap.xml that lists all important pages with accurate metadata and no broken or conflicting URLs.

Why does it matter?

Sitemaps are critical for SEO because they:

  • Improve Crawlability: Help search engines discover pages that might not be found through internal links alone, especially "orphan pages"
  • Enhance Indexability: Ensure important content is considered for indexing by providing direct URL signals
  • Support Rankings: Enable faster discovery and indexing of new or updated content
  • Optimize Crawl Budget: Guide crawlers to priority pages efficiently, especially for large sites

Sitemaps directly contribute to the overall SEO health score by ensuring search engines can efficiently discover and index your content. Sites without sitemaps or with broken sitemaps may experience delayed indexing and reduced visibility in search results.

How to fix it

  1. Ensure sitemap accessibility: Place a sitemap file at your root domain (/sitemap.xml) and ensure it returns a 200 OK status without redirects.

  2. Declare sitemap location: Add a Sitemap: directive in your robots.txt file pointing to your sitemap URL.

  3. Remove conflicting URLs: Exclude URLs from your sitemap that are blocked by robots.txt or marked with noindex tags.

  4. Split large sitemaps: If your sitemap exceeds 50,000 URLs or 50MB, split it into multiple sitemaps and use a sitemap index file.

  5. Maintain canonical consistency: Ensure every URL in your sitemap matches the canonical URL exactly (same scheme, domain, path, and trailing slash).

  6. Include media content: For important images and videos, use image and video sitemap extensions to help search engines discover this content.

  7. Validate regularly: Periodically check that all URLs in your sitemap return 200 OK status codes and aren't blocked by robots.txt or noindex directives.

Examples

Example 1: Missing Sitemap

Problematic State (Fails): A website has no sitemap file at /sitemap.xml and no Sitemap: directive in robots.txt. Search engines must discover all pages through internal links, which may miss orphan pages.

Corrected State (Passes): Create a sitemap at https://example.com/sitemap.xml and add Sitemap: https://example.com/sitemap.xml to your robots.txt file.

Example 2: Sitemap with Broken URLs

Problematic State (Fails): A sitemap contains URLs that return 404 errors:

<url>
  <loc>https://example.com/old-page</loc>
</url>

The URL https://example.com/old-page returns a 404 Not Found status.

Corrected State (Passes): Remove broken URLs from the sitemap or fix the pages so they return 200 OK:

<url>
  <loc>https://example.com/new-page</loc>
</url>

Example 3: Sitemap with Conflicting Signals

Problematic State (Fails): A URL is in the sitemap but blocked by robots.txt:

<!-- sitemap.xml -->
<url>
  <loc>https://example.com/private-page</loc>
</url>

# robots.txt
Disallow: /private-page

Corrected State (Passes): Either remove the URL from the sitemap or remove the Disallow rule:

<!-- sitemap.xml -->
<url>
  <loc>https://example.com/private-page</loc>
</url>

# robots.txt
# Allow crawling of private-page (but keep it noindex if needed)

How PixyScan detects this

PixyScan performs comprehensive sitemap validation through the following logical steps:

  1. Sitemap Discovery: PixyScan looks for your sitemap by checking robots.txt for Sitemap: directives, trying common paths like /sitemap.xml, and examining HTML <link> tags in your pages.

  2. Sitemap Fetching: If a sitemap is found, PixyScan fetches the file and determines whether it's a standard sitemap or a sitemap index file. For index files, PixyScan fetches each child sitemap recursively.

  3. URL Validation: For each URL listed in the sitemap, PixyScan checks:

    • The HTTP status code (should be 200 OK)
    • Whether the URL is blocked by robots.txt
    • Whether the URL has a noindex directive
    • Whether the canonical URL on the page matches the sitemap URL exactly

  4. Issue Identification: PixyScan raises issues when:

    • No sitemap is discovered (CRITICAL)
    • The sitemap returns a non-200 status code (CRITICAL)
    • URLs in the sitemap are blocked or have noindex tags (WARNING)
    • Canonical URLs don't match sitemap URLs (WARNING)
    • The sitemap isn't declared in robots.txt (SUGGESTION)

References