Show HN: TurboQuant for mlx-lm (Apple Silicon) (github.com)

A new standalone TurboQuant adapter for mlx-lm has been released, specifically optimized for Apple Silicon. This adapter facilitates non-uniform quantization using a custom Metal kernel, allowing the efficient, calibration-free compression of neural network weights and key-value caches without compromising model performance. Leveraging a Randomized Hadamard Transform, TurboQuant can distribute outliers across vector coordinates, enabling low-bit quantization methods to function more effectively. With its drop-in TurboQuantKVCache, it maintains inner product calculations across rotated vectors, crucial for on-device applications. The significance of this adapter lies in its ability to dramatically improve both speed and memory efficiency for AI models on Apple hardware. Benchmark results indicate that it achieves up to 2.1 times faster decoding compared to baseline fp16 configurations, while also reducing the model's memory footprint considerably—from 3.44 GB to 1.42 GB for weights and down to just 0.066 GB for the KV cache. Additionally, it enhances model usability at lower bit rates, making it feasible to run sophisticated AI tasks on resource-constrained devices while preserving near-fp16 quality. This advancement is particularly relevant for developers looking to optimize AI applications for Apple Silicon, highlighting TurboQuant's potential impact on the AI/ML community's ongoing pursuit of efficiency in model deployment.