Cutting LLM token Usage by ~80% using REPL driven document analysis (yogthos.net)

A new tool named Matryoshka has been introduced to enhance document analysis for AI agents by cuttings token usage by over 80%. Traditional approaches to analyzing large codebases require significant token consumption, making tasks cumbersome and costly. Matryoshka addresses this challenge by caching analysis results and allowing for interactive queries without repeatedly sending the same document content. This approach allows AI models like Claude to treat documents as external knowledge bases, facilitating more efficient information retrieval and reducing context degradation issues commonly seen in lengthy inputs. Matryoshka integrates two key innovations: it uses a declarative query language for task-specific exploration, which improves clarity and efficiency, and it maintains a persistent analytical state where only relevant results are processed. This method not only saves on computational resources but also enables LLMs to focus on novel inquiries rather than redundant data. The system's hybrid approach, which combines full reads of smaller files with selective queries of larger ones, ensures comprehensive coverage while retaining optimal performance. Real-world applications, such as analyzing the anki-connect codebase, demonstrate significant improvements in efficiency and effectiveness for AI-driven document analysis tasks.