Skip to content
Browse all guides

Issue #186 · standard

Issue 186

What is this issue?

llms.txt is a plain-text Markdown file placed at the root of a website (e.g., https://example.com/llms.txt). It helps AI language models discover and understand a site's key content in a structured, LLM-friendly format. This issue checks whether your site has an llms.txt file, whether it's properly formatted, and accessible.

A passing implementation requires:

  • An llms.txt file exists at your root domain
  • The file returns HTTP 200 OK with Content-Type: text/plain
  • The file follows proper Markdown structure (H1 heading, blockquote description, section headings)
  • All listed URLs in the file return 200 OK status
  • The file size is under the recommended limit (50-100KB)
  • A discovery tag is present in your homepage <head>

Example: A properly configured llms.txt file at https://example.com/llms.txt with a site title, description, section headings, and links to key pages.

Why does it matter?

The llms.txt file is important for SEO because it:

  • Improves AI Discoverability: Helps AI language models find and understand your most important content
  • Enhances AI Search/AEO: Increases chances of your content being referenced in AI-generated answers
  • Provides Structured Context: Gives LLMs a curated index of your site instead of parsing entire HTML pages
  • Future-Proofs SEO: As AI search grows, having an llms.txt file positions your site for AI-driven discovery

While still a proposed standard (not yet officially adopted by all AI models), llms.txt is increasingly used by technical sites and AI platforms. Implementing it early can give you an advantage in AI search visibility.

Resolving this issue improves your overall SEO health score by ensuring your site is optimized for the growing AI search ecosystem.

How to fix it

  1. Create the file: Create a plain text file named exactly llms.txt at your domain root (https://example.com/llms.txt).

  2. Structure with Markdown: Format the file using proper Markdown:

    • Start with an H1 heading: # Your Site Name
    • Add a blockquote description: > Brief description of your site
    • Add section headings: ## Documentation, ## Products, etc.
    • Add links: - [Page Title](https://example.com/page): Description

  3. Serve with correct headers: Ensure the file returns:

    • HTTP 200 OK status
    • Content-Type: text/plain header

  4. Validate links: Ensure all URLs listed in the file return 200 OK when accessed.

  5. Add discovery tag: Include this tag in your homepage <head>:

    <link type="text/plain" rel="llms" href="/llms.txt" />

  6. Keep file size reasonable: Aim for under 50KB to ensure fast processing by LLMs.

  7. Test the file: Validate that:

    • The file is accessible at the root URL
    • The Markdown is properly formatted
    • All listed links work
    • The discovery tag is present in your homepage

Examples

Example 1: Missing llms.txt file

Problematic state (fails):

  • No llms.txt file exists at https://example.com/llms.txt
  • PixyScan returns a 404 error when attempting to fetch the file
  • Issue raised: "llms.txt file is missing"

Corrected state (passes):

  • Create llms.txt file at the root domain
  • File returns HTTP 200 OK
  • Issue resolved

Example 2: Improperly formatted llms.txt

Problematic state (fails):

My Website
This is my website description.
Some links:

- https://example.com/page1
- https://example.com/page2

Issues:

  • Missing H1 heading (#)
  • Missing blockquote description (>)
  • No section headings (##)
  • URLs not formatted as Markdown links

Corrected state (passes):

# My Website

> This is my website description.

## Documentation

- [Getting Started](https://example.com/docs): Learn how to use our product
- [API Reference](https://example.com/api): Complete API documentation

## Products

- [Product A](https://example.com/product-a): Our flagship product

Example 3: Missing discovery tag

Problematic state (fails):

  • llms.txt file exists and is properly formatted
  • Homepage HTML does not include the discovery tag
  • Issue raised: "Discovery tag is missing from homepage"

Corrected state (passes): Add to homepage <head>:

<link type="text/plain" rel="llms" href="/llms.txt" />

PixyScan detects the tag and the issue is resolved.

How PixyScan detects this

PixyScan detects this issue by fetching the file directly — it requests https://<origin>/llms.txt (the origin is derived from the scan's seed URL) and inspects the response. It does not look for any llms.txt: directive in robots.txt; no such directive exists in RFC 9309 or in the llms.txt proposal (llmstxt.org), so the only way to know whether the file exists is to request it.

Detection steps (function checkLlmsTxtFile in apps/crawler/seo-audit-checks.js, run once per scan by the GEO & AI Engine Signals toggle group):

  1. Fetch: GET https://<origin>/llms.txt with a 10-second timeout. All HTTP status codes are accepted for inspection (no exception is thrown on 4xx/5xx).

  2. Existence check: the file counts as present only when the response passes a plausibility test:

    • HTTP status is 200
    • the body is a non-empty string
    • the Content-Type header is not text/html
    • the body does not start with <!doctype or <html

  3. Soft-404 guard: the content-type/body sniffing above exists because SPA routers and custom error pages often answer 200 with the HTML app shell for every path. Such a response is classified as a soft-404 — the URL answers 200 but the file does not actually exist.

  4. Issue identification: the STANDARD-severity finding is raised when the file is absent:

    • non-200 status → detail status: "not-found" (message includes the HTTP status)
    • 200 but HTML/soft-404 → detail status: "soft-404"

  5. Network errors are not findings: if the request itself fails (DNS failure, timeout, TLS error), PixyScan could not ask the question — that is not evidence the file is missing. No issue is raised; the error is returned to the caller for logging only.

There is no format validation, link validation, size check, or homepage discovery-tag check — existence of a plausible plain-text file at /llms.txt is the entire check. This is a site-wide check recorded once per scan, not per page.

References