Show HN: Fast NF4 dequantization Triton kernel (1.41x faster than bitsandbytes) (github.com)

A new Triton kernel for the NF4 (NormalFloat 4-bit) dequantization process has been developed, achieving speed improvements of 1.27x to 1.72x over the existing bitsandbytes C++ implementation. The kernel optimizes the dequantization workflow crucial for large language models (LLMs) by performing multiple operations—bit unpacking, NF4 table lookup, and scaling—within a single GPU pass, significantly reducing CPU dispatch overhead. This implementation was tested on an RTX 3050 Laptop GPU, showing remarkable results across various tensor sizes, with notable improvements in processing time. This advancement is significant for the AI and machine learning community as it enhances the efficiency of memory usage in LLMs, particularly while using 4-bit quantization to fit weights in GPU memory. The transition from C++ to Triton allows for finer control over GPU resources, utilizing inline PTX assembly to eliminate unnecessary memory reads and improve performance consistency. The kernel also accommodates dynamic shapes, making it versatile for various applications. The project is available on GitHub, encouraging adoption and further optimization within the community.