Show HN: Sparse kernel delivering 36% higher training throughput (huggingface.co)

A new code repository by Jakub Ćwirlej has introduced a sparse kernel designed for transformer architectures, achieving 36% higher training throughput compared to traditional dense feed-forward networks (FFNs). This library combines Triton GPU kernels with a PyTorch wrapper, facilitating sparse SwiGLU layers that skip inactive neurons during both forward and backward passes. The implementation is targeted at models with 306 million parameters, where only 6.25% of the FFN neurons remain active at any given time, significantly enhancing efficiency while retaining model performance. This development is significant for the AI and machine learning community as it addresses the computational bottlenecks prevalent in deep learning training, particularly in transformer models, by optimizing memory and compute resources. While the results are promising, the kernel is still experimental, and its effectiveness when training models from scratch remains uncertain. Researchers are encouraged to explore this approach for sparse feed-forward layers in transformer training, with the potential to unlock faster training times and reduce energy consumption in AI model development. The library, however, is restricted to non-commercial use and requires validation for specific GPU configurations.