How scanning works

Crawl starts at the site address and follows internal links, plus anything in your sitemaps. Use it for a full picture.

Custom takes an explicit list of URLs and checks only those. Use it when you have shipped a handful of pages and want them checked without re-crawling everything.

• robots.txt, including Disallow rules and Crawl-delay.
• Your sitemaps, which it reads to discover URLs and to check that canonicals actually appear in them.
• Your page cap and exclude patterns.

The crawler identifies itself with a pixyscan user agent, so you can allow or rate-limit it specifically.

PixyScan reads the HTML your server returns. It does not execute JavaScript.

This is deliberate rather than a limitation to work around: a search crawler sees that HTML first, and several checks exist precisely to catch metadata that only appears once JavaScript has run. If your title or description is injected client-side, PixyScan reports it as missing from the server-rendered HTML — which is the problem, not a false positive.

If a URL you expected is not in the results, it is almost always one of:

• The page cap was reached before the crawler got to it.
• An exclude pattern matched it.
• robots.txt disallows it.
• Nothing links to it and it is in no sitemap — in which case PixyScan reports it as an orphan page when it does find it another way.
• It returned a non-200 status, which shows up under crawl behaviour rather than as a page report.