A 3-Layer Cache Architecture Cuts LLM API Costs by 75% (github.com)

A new three-layer caching architecture called Distributed Semantic Cache has been introduced, significantly reducing LLM API costs by up to 75%. Traditional caching methods only capture 20-30% of repeated queries due to variations in user phrasing, leading to high operational costs—up to $30K a month for 1 million queries on models like GPT-4. The innovative structure comprises three layers: an Exact Match layer for instant hits, a Normalized Match layer that captures variations in user queries, and a Semantic Match layer that uses embedding similarity for broader contextual matches. The combination handles 60-75% of requests efficiently, drastically cutting API calls to a mere 300,000 and lowering monthly expenses to approximately $9,000. This approach leverages complex techniques like HNSW for approximate nearest neighbor searches, balancing speed and recall, alongside advanced normalization strategies that boost hit rates without the associated costs of full embeddings. The system efficiently compresses query data while maintaining performance metrics, offering scalability with various storage options and robust privacy modes. By streamlining query processing and optimizing cache management, this architecture emerges as a significant advancement for the AI/ML community, especially for applications relying on large language models, ultimately enhancing user experience while minimizing operational costs.