Llmtext – an open source toolkit for llms.txt adoption (llmtext.com)

Llmtext is an open‑source toolkit that helps sites publish llms.txt—an emerging, robots.txt‑like standard that gives AI agents a compact, AI‑friendly table of contents pointing to plain‑text or Markdown pages. The toolkit includes a CREATE tool (which uses a Parallel Extract API to scrape HTML and auto‑convert pages to Markdown), a CHECK validator against the llmstxt.org spec, and an MCP (Model Context Protocol) server that turns any valid llms.txt into a structured resource Claude Desktop and other MCP‑enabled assistants can fetch directly. That flow reduces costly, error‑prone full‑site scraping, lowers latency and bandwidth, and significantly cuts hallucination risk by letting agents read canonical text versions of documentation and docs pages. Key technical rules and implications: llms.txt should live at your domain root (or per‑subdomain), link only to text/markdown URLs (not HTML), and remain compact—the llms.txt file and each linked page should be under ~10k tokens for efficient context use. MCP servers fetch content dynamically (no reinstall needed when content changes). Unlike vector/RAG search, llms.txt provides explicit document structure and navigation, which is ideal for well‑organized docs; RAG still helps with unstructured corpora. Early adoption gives sites “agent search optimization” benefits as AI traffic grows, and the toolkit provides install instructions (installthismcp.com) and compatibility checks to streamline integration.