Show HN: Sulci – a semantic cache that knows when not to answer (github.com)

Sulci Cache has introduced a novel semantic caching library designed to enhance cost-efficiency and performance for applications using large language models (LLMs). This drop-in Python tool caches LLM responses based on semantic content rather than exact string matches. For instance, if one user asks about deploying to AWS and another poses a similar question, Sulci Cache retrieves the existing answer from its cache, significantly reducing the need for repeated LLM calls and cutting down response times to around 87 ms from a typical 1-3 seconds for new API calls. This development is significant for the AI/ML community as it addresses the dual challenges of rising API costs and latency in LLM applications. With an impressive 99.9% hit rate for repeat questions and nearly 94% accuracy for cache responses, Sulci Cache not only optimizes operational costs but also enhances user experience by providing quick and relevant answers. Its context-aware capabilities allow it to blend recent conversation snippets into its search, improving the relevance of cached data and potentially reshaping how LLM applications manage and retrieve information.