Skip to content
Browse all guides

Issue #166 · important

Issue 166

What is this issue?

This issue checks whether a website's internal search results pages are blocked from search engine crawlers using the robots.txt file or meta robots tags, preventing crawl budget waste and duplicate content issues.

A passing implementation means:

  • Search results pages (e.g., /search?q=..., /results?...) are disallowed in robots.txt OR have noindex meta tags
  • The block is properly implemented and covers all search result URL patterns
  • Faceted search and filtered results pages are also blocked if applicable

Example robots.txt:

User-agent: *
Disallow: /search
Disallow: /results
Disallow: /search-results

Example meta robots tag:

<meta name="robots" content="noindex, nofollow" />

Why does it matter?

Crawl Budget Waste: Search results pages often generate infinite URL variations with minimal unique content. Crawling them wastes valuable crawl budget that should be spent on important pages.

Duplicate Content: Search results pages often contain snippets of content from other pages, creating near-duplicate content that can confuse search engines.

Poor User Experience in SERPs: Search results pages in search results provide a poor user experience - users expect to land on actual content, not more search results.

Thin Content: Search results pages are typically thin content pages that don't provide value to search engine users.

SEO Health Score: Blocking search results pages improves the technical SEO score by preventing crawl budget waste and duplicate content issues.

How to fix it

  1. Identify search result URLs - Determine the URL patterns for your site's search functionality (e.g., /search?q=, /results?, /search-results).

  2. Block in robots.txt - Add Disallow directives for search result URL patterns:

    Disallow: /search
Disallow: /results

  3. Alternative: Use meta robots - If you can't block in robots.txt, add <meta name="robots" content="noindex"> to search result pages.

  4. Check faceted navigation - If your site has faceted search (filters, sorting), ensure those URLs are also blocked:

    Disallow: /search?
Disallow: /products/filter/

  5. Test the block - Use Google Search Console's robots.txt tester to verify the block is working.

  6. Monitor indexing - Check Google Search Console to ensure search result pages are not appearing in the index.

Examples

Example 1: Basic Search Results Block

Problematic State (Fails): Search results pages are crawlable:

https://example.com/search?q=product

Returns 200 OK and is indexed by search engines.

Corrected State (Passes): Block in robots.txt:

User-agent: *
Disallow: /search

Example 2: Faceted Search URLs

Problematic State (Fails): Faceted navigation creates infinite URLs:

https://example.com/products?color=red&size=large&brand=nike
https://example.com/products?size=large&brand=nike&color=red

These create duplicate content issues.

Corrected State (Passes): Block parameterized URLs:

User-agent: *
Disallow: /products?

Example 3: Using Meta Robots

Problematic State (Fails): Can't block in robots.txt, but search pages are indexed.

Corrected State (Passes): Add noindex meta tag to search result pages:

<head>
  <meta name="robots" content="noindex, nofollow" />
</head>

How PixyScan detects this

PixyScan performs the following checks:

  1. URL Pattern Detection: PixyScan identifies search result pages by looking for common URL patterns:

    • /search?q=...
    • /results?...
    • /search-results
    • Other custom search URL patterns

  2. Robots.txt Check: PixyScan checks if these URL patterns are disallowed in robots.txt.

  3. Meta Robots Check: For pages that are accessible, PixyScan checks if they have noindex meta tags.

  4. Faceted Navigation Check: PixyScan also checks for faceted search URLs with multiple parameters.

  5. Issue Identification: PixyScan raises issues when:

    • Search result pages are not blocked in robots.txt (WARNING)
    • Search result pages don't have noindex tags (WARNING)
    • Faceted navigation creates infinite URL variations (SUGGESTION)

References