Show HN: Mdlens – Reduce token spend and boost retrieval on Markdown-heavy repos (github.com)

A new command-line interface tool called Mdlens has been introduced, streamlining the process of retrieving information from Markdown-heavy repositories. The standout feature, `mdlens scout`, allows users to input a directory and a natural language question, yielding a structured evidence pack that highlights relevant files, section IDs, and snippets. This tool replaces conventional search patterns by efficiently navigating various Markdown document types, helping users quickly pinpoint answers without sifting through extensive file reads. Mdlens is significant for the AI/ML community as it optimizes token usage during query processing, essential for minimizing costs associated with AI language models. In evaluations comparing Mdlens to traditional methods, it demonstrated improved accuracy and efficiency, achieving an average success rate of 22.7 out of 30 tasks while reducing the number of tool calls and dramatically lowering operational costs. The project underscores a growing trend towards specialized tools that leverage AI capabilities to enhance information retrieval, particularly in documentation-heavy contexts, illustrating potential advancements for AI agents in managing and utilizing Markdown resources effectively.