Compaction can save 50% due to cache-read cost (relace.ai)

A recent announcement from Relace reveals a new compaction model that can potentially cut coding workload costs by 50% by optimizing cache-read expenses during AI coding sessions. As companies face soaring AI costs, particularly around token usage, this approach focuses on compacting inactive user agent traces, significantly reducing the quadratic growth in cache-read costs. The model operates at over 50,000 tokens per second, allowing for seamless integration without impacting user experience. The significance of this development lies in addressing the inefficiencies of long-running conversations in AI agent systems. The findings indicate that input tokens, rather than output tokens, have become the primary source of expense. By compacting data on cache misses, the method transforms cache read costs from quadratic to linear, leading to substantial savings as sessions accumulate. Testing has shown that even a conservative compaction strategy can yield over 56% savings on token costs, positioning this approach as a simple yet powerful optimization strategy for AI/ML applications, especially in coding environments.