Skip to content
Browse all guides

Issue #106 · important

Issue 106

What is this issue?

This issue checks whether your pages have a properly configured <meta name="robots"> tag with specific directives that control how search engines display your content in search results.

A passing implementation includes a meta robots tag with the following recommended settings:

  • max-snippet:-1 (no limit on snippet length)
  • max-image-preview:large (allow large image thumbnails)
  • max-video-preview:-1 (no limit on video preview duration)

Example of correct implementation:

<meta
  name="robots"
  content="max-snippet:-1, max-image-preview:large, max-video-preview:-1"
/>

Without this tag, search engines may use default settings that could limit how your content appears in SERPs.

Why does it matter?

The meta robots tag directly impacts how your pages appear in search engine results pages (SERPs):

  • Rich snippets: The max-snippet directive controls how much text content can appear in search snippets. Setting it to -1 allows search engines to show as much as they deem relevant.

  • Visual appeal: The max-image-preview:large directive enables large thumbnail images next to your search results, which can significantly increase click-through rates.

  • Video visibility: The max-video-preview directive controls whether video content can show preview clips in search results, helping video content stand out.

  • CTR improvement: Properly configured meta robots tags can make your search listings more visually appealing and informative, leading to higher click-through rates.

Resolving this issue improves your SEO health score by ensuring your content has the best possible presentation in search results, which can lead to increased organic traffic.

How to fix it

Follow these steps to implement the meta robots tag correctly:

  1. Add the meta tag to your HTML head section: Place the following line inside the <head> section of your HTML document:

    <meta
  name="robots"
  content="max-snippet:-1, max-image-preview:large, max-video-preview:-1"
/>

  2. For specific page types: You can adjust the values based on your needs:

    • For pages where you want to limit snippets: Use max-snippet:50 (limits to 50 characters)
    • For pages with sensitive images: Use max-image-preview:standard instead of large
    • For pages without video: The max-video-preview directive won't have any effect

  3. Verify implementation: Use View Page Source or browser developer tools to confirm the tag appears correctly in the rendered HTML.

  4. Test with search engines: Use Google Search Console's URL Inspection tool to see how Google interprets your robots meta directives.

Note: This tag should be present on all indexable pages of your website.

Examples

Example 1: Correct Implementation

Scenario: A properly configured page with meta robots tag.

Correct State (Passes):

<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <meta
    name="robots"
    content="max-snippet:-1, max-image-preview:large, max-video-preview:-1"
  />
  <title>Page Title</title>
</head>

Example 2: Missing Meta Robots Tag

Scenario: Page has no meta robots tag.

Problematic State (Fails):

<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <title>Page Title</title>
</head>

Why it fails: Without the meta robots tag, search engines may use default settings that limit how your content appears in search results.

Corrected State (Passes):

<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <meta
    name="robots"
    content="max-snippet:-1, max-image-preview:large, max-video-preview:-1"
  />
  <title>Page Title</title>
</head>

Example 3: Incomplete Meta Robots Tag

Scenario: Meta robots tag is present but missing required directives.

Problematic State (Fails):

<head>
  <meta name="robots" content="max-snippet:-1" />
  <title>Page Title</title>
</head>

Why it fails: Missing max-image-preview and max-video-preview directives, which may limit how images and videos appear in search results.

Corrected State (Passes):

<head>
  <meta
    name="robots"
    content="max-snippet:-1, max-image-preview:large, max-video-preview:-1"
  />
  <title>Page Title</title>
</head>

Example 4: Restrictive Meta Robots Settings

Scenario: Page uses restrictive settings that limit search result appearance.

Problematic State (Fails):

<head>
  <meta
    name="robots"
    content="max-snippet:0, max-image-preview:none, max-video-preview:0"
  />
  <title>Page Title</title>
</head>

Why it fails: These settings prevent search engines from showing snippets, images, and video previews, which can significantly reduce click-through rates.

Corrected State (Passes):

<head>
  <meta
    name="robots"
    content="max-snippet:-1, max-image-preview:large, max-video-preview:-1"
  />
  <title>Page Title</title>
</head>

How PixyScan detects this

PixyScan performs the following checks to detect this issue:

  1. HTML parsing: The crawler extracts the <head> section from the page's HTML content.

  2. Meta tag search: It looks for a <meta> tag with the attribute name="robots".

  3. Content validation: If the tag is found, PixyScan checks the content attribute for the presence of:

    • max-snippet directive (value can be -1 or a number)
    • max-image-preview directive (value should be large, standard, or none)
    • max-video-preview directive (value can be -1 or a number)

  4. Pass/Fail determination:

    • Passes: If all three directives are present in the meta robots tag content
    • Fails: If the meta robots tag is missing, or if any of the three directives are missing

The detection focuses on whether the tag exists and contains the recommended directives, not on the specific values used.

References