Issue 165

The Crawl-delay directive in robots.txt tells web crawlers how many seconds to wait between requests to your website. This issue checks whether a crawl-delay directive is properly set when your server needs rate limiting to prevent overload from web crawlers.

A passing implementation requires:

• A Crawl-delay directive is present in your robots.txt file
• The directive is set to a positive numeric value (e.g., Crawl-delay: 1)
• The directive is placed in the correct User-agent group
• Your robots.txt file is accessible and returns 200 OK

Example: A robots.txt file with Crawl-delay: 1 in the User-agent: * section to tell all crawlers to wait 1 second between requests.

The crawl-delay directive is important for SEO and server stability because it:

• Protects Server Resources: Prevents crawlers from overwhelming your server with too many simultaneous requests
• Maintains Site Performance: Ensures your website remains fast for real users even when being crawled
• Supports Crawl Budget Management: Helps search engines crawl your site more efficiently without causing server strain
• Prevents Downtime: Reduces the risk of server crashes due to aggressive crawling

While not all crawlers respect the crawl-delay directive, it's a helpful signal for crawlers that do support it. This indirectly supports your SEO health score by maintaining server stability and ensuring consistent crawlability.

1. Check if you need rate limiting: Evaluate whether your server experiences high load during crawls or has limited resources.

2. Edit your robots.txt file: Add a Crawl-delay directive to your robots.txt file:

   User-agent: *
   Crawl-delay: 1
   

   (The number represents seconds to wait between requests)

3. Choose an appropriate value: Set a reasonable delay based on your server capacity:

   • 1 second for most sites
   • 2-5 seconds for servers with limited resources
   • Higher values for very resource-constrained servers

4. Place in correct section: Ensure the Crawl-delay directive is within a User-agent group (either * for all crawlers or a specific crawler like Googlebot).

5. Test your robots.txt: Verify your robots.txt file is accessible and properly formatted.

6. Monitor server load: After implementing, monitor your server performance during crawls to ensure the delay is effective.

Example 1: Adding Crawl-delay

Problematic State (Fails): Server experiences high load during crawls, but no crawl-delay is set:

User-agent: *
Disallow: /admin/

Corrected State (Passes): Add Crawl-delay directive:

User-agent: *
Crawl-delay: 1
Disallow: /admin/

Example 2: Wrong Value

Problematic State (Fails): Crawl-delay set to an invalid value:

Crawl-delay: yes

Corrected State (Passes): Use a numeric value (seconds):

Crawl-delay: 1

Example 3: Specific Crawler

Problematic State (Fails): Crawl-delay is set for all crawlers, but you only want to limit a specific crawler:

User-agent: *
Crawl-delay: 1

Corrected State (Passes): Target specific crawler:

User-agent: Bingbot
Crawl-delay: 2

User-agent: *
Disallow: /admin/

PixyScan performs crawl-delay directive checks through the following logical steps:

1. Configuration Check: PixyScan checks if your site is configured as requiring rate limiting (based on server capacity settings).

2. Robots.txt Fetch: PixyScan attempts to fetch your robots.txt file from the root domain.

3. Directive Parsing: If robots.txt exists, PixyScan parses the file line-by-line to find Crawl-delay directives within User-agent groups.

4. Value Validation: PixyScan validates that:

   • A Crawl-delay directive exists
   • The value is a positive number
   • The directive is in the correct section

5. Issue Identification: PixyScan raises issues when:

   • Crawl-delay is missing but server needs rate limiting (SUGGESTION)
   • The value is invalid (not a number) (WARNING)
   • The directive is in the wrong section (WARNING)