Sparser, Faster, Lighter Transformer Language Models (pub.sakana.ai)

Researchers from Sakana AI, in collaboration with NVIDIA, have developed new sparse data structures and GPU kernels aimed at enhancing the efficiency of large language models (LLMs) during both inference and training. This approach seeks to exploit the unstructured sparsity within the feedforward layers of LLMs, which traditionally consume a disproportionate amount of computational resources. By applying L1 regularization, they demonstrated that up to 95% of the hidden activations in these layers can be made sparse with minimal impact on model performance. Their advanced kernel implementations demonstrated a significant over 20% increase in processing speed on NVIDIA H100 GPUs while also reducing memory usage and energy consumption. This innovation marks a critical advancement for the AI/ML community, as it directly addresses the pressing challenge of scaling LLMs efficiently without compromising performance. Leveraging a new sparse data format called TwELL, the researchers optimized the interaction between these sparse activations and the high-performance matrix multiplication capabilities of modern GPUs, minimizing the global memory accesses that typically create bottlenecks in such computations. This work not only promises to enhance the practical deployability of LLMs in resource-constrained environments but also lays the foundation for future research into even more efficient AI model architectures.