Issue 164

This issue checks whether a website's robots.txt file includes a Sitemap: directive that explicitly declares the location of the XML sitemap, providing an additional method for search engines to discover the sitemap.

A passing implementation means:

• The robots.txt file exists at /robots.txt
• It contains one or more Sitemap: directives pointing to valid sitemap URLs
• The sitemap URLs are absolute and properly formatted
• Multiple sitemaps (if applicable) are all listed

Example:

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-images.xml

Sitemap Discovery: While search engines can discover sitemaps through other means (Google Search Console, sitemap index files), the Sitemap: directive in robots.txt is a standard method recognized by all major crawlers including Google, Bing, Yahoo, and others.

Crawler Efficiency: Search engines often check robots.txt first before crawling. Having the sitemap directive there allows them to discover and process the sitemap immediately.

Backup Discovery Method: If sitemaps are not submitted through Search Console or are not linked from the homepage, the robots.txt directive serves as a fallback discovery mechanism.

SEO Best Practice: Including sitemap locations in robots.txt is a widely recommended SEO practice that demonstrates technical optimization.

SEO Health Score: Having the Sitemap: directive improves the technical SEO score by showing comprehensive sitemap accessibility.

1. Locate your robots.txt file - This should be at the root of your domain (https://example.com/robots.txt).

2. Add Sitemap directive(s) - Add one line per sitemap at the end of the robots.txt file:

   Sitemap: https://example.com/sitemap.xml
   

3. Use absolute URLs - Always use the full absolute URL including https://, not relative paths.

4. List all sitemaps - If you have multiple sitemaps (main, images, videos, etc.), list each one on a separate line.

5. Validate the format - Ensure there are no spaces before Sitemap: and the colon is immediately followed by the URL.

6. Test accessibility - Verify the robots.txt file is accessible and properly formatted by visiting https://example.com/robots.txt in a browser.

Example 1: Missing Sitemap Directive

Problematic State (Fails): robots.txt exists but doesn't declare the sitemap:

User-agent: *
Disallow: /admin/

Corrected State (Passes): Add the Sitemap directive:

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Example 2: Multiple Sitemaps

Problematic State (Fails): Only the main sitemap is declared, but image sitemap is missing:

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Corrected State (Passes): Declare all sitemaps:

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-images.xml

Example 3: Relative URL

Problematic State (Fails): Using relative URL instead of absolute:

Sitemap: /sitemap.xml

Corrected State (Passes): Use absolute URL:

Sitemap: https://example.com/sitemap.xml

PixyScan performs the following checks:

1. Fetches robots.txt - The crawler requests the /robots.txt file from the domain being audited.

2. Parses the content - PixyScan reads through the robots.txt file line by line.

3. Looks for Sitemap directives - The system searches for lines starting with Sitemap: (case-insensitive).

4. Validates the URLs - For each Sitemap: directive found, PixyScan:

   • Checks that the URL is absolute (starts with http:// or https://)
   • Attempts to fetch the sitemap to verify it's accessible
   • Validates the sitemap format

5. Issue Identification: PixyScan raises issues when:

   • No Sitemap: directive is found in robots.txt (SUGGESTION)
   • The sitemap URL is relative instead of absolute (WARNING)
   • The sitemap URL returns an error (WARNING)
   • The Sitemap: directive is malformed (WARNING)