llms.txt

llms.txt is a community-proposed standard for a plain-text file placed at the root of a website (e.g., yoursite.com/llms.txt) that communicates structured information about the site to AI crawlers, large language models, and RAG systems. The file serves a similar conceptual role to robots.txt (crawler instructions) and sitemap.xml (content index), but is specifically designed for the needs of AI content ingestion rather than traditional search.

The standard, popularized by Answer.AI's Jeremy Howard in 2024, specifies that the file should contain a brief description of what the site is, who it is for, and what its main content areas are. It can also include links to key documentation, content types, and specific pages that LLMs should prioritize when building knowledge of the domain. An extended version, llms-full.txt, can include the full text of documentation in LLM-optimized format.

For developers and technical content sites, llms.txt is particularly valuable because it allows the site to present its most important information in a format already optimized for LLM ingestion — plain text, clearly structured, without navigation chrome or JavaScript that might confuse parsing. Several major documentation platforms and developer tool companies were early adopters, recognizing that developers increasingly use LLMs to answer questions about tools and APIs.

While not yet an official standard ratified by any standards body, llms.txt has gained significant adoption across the developer and AI-native content ecosystem. AI SEO practitioners recommend its adoption as a low-cost, high-upside technical optimization that signals AI-readiness to crawlers and improves content ingestion quality.

A developer tools company creates an llms.txt file listing its API documentation sections, SDK guides, and changelog pages — resulting in ChatGPT and Claude providing more accurate answers about the tool because they have a structured content map to reference.