Introducing RadixAttention to Trellis (trellis.unfoldml.com)

Trellis has introduced RadixAttention, an innovative optimization for LLM inference, aimed at enhancing performance while ensuring user data privacy. This system allows users to deploy LLM technology on their existing hardware, such as laptops and workstations, and significantly improves the prefill phase—a compute-heavy process essential for generating subsequent tokens in chat-based applications. The new caching strategy builds upon two key observations: the ability to cache embeddings of previous tokens, and the frequent reuse of common prompt prefixes in LLM sessions. RadixAttention utilizes a radix tree data structure to efficiently store token embeddings, minimizing memory usage by eliminating redundancy in shared strings. Benchmarks show a performance improvement of 30-40% in both throughput and memory allocation when multiple requests share similar prefixes, leading to faster time-to-first-token generation. As the demand for longer LLM sessions grows, these optimizations could further enhance the efficiency of LLM applications, making RadixAttention a significant advancement in the AI/ML community’s pursuit of scalable, privacy-respecting inference solutions.