Prompt caching: 10x cheaper LLM tokens, but how? (ngrok.com)

Recent developments in prompt caching have led to a significant breakthrough in the efficiency of language models, making input tokens for OpenAI and Anthropic's APIs ten times cheaper. This technique not only reduces costs but also dramatically decreases latency—claiming up to an 85% reduction for longer prompts. During extensive testing, prompt caching demonstrated faster time-to-first-token latency when all input tokens were cached, boosting the interaction speed for users. The significance of prompt caching lies in its deep integration with the architecture of large language models (LLMs). It operates primarily within the attention mechanism of transformers, where embeddings of input tokens are processed. While it doesn’t save the entire output, it intelligently manages the input representations, reusing them across requests to optimize the processing time and resources. As LLMs continue to grow in complexity and application, techniques like prompt caching are becoming essential for improving their accessibility and responsiveness, paving the way for more interactive and cost-effective AI applications.