Mlx-optiq: per-layer mixed-precision LLM quantization for Apple Silicon (mlx-optiq.com)

Mlx-optiq has been launched as a versatile toolkit for quantizing, fine-tuning, and serving large language models (LLMs) entirely on Apple Silicon, from M1 to M5. This innovation allows users to run powerful LLMs locally without the need for GPUs or API keys, making advanced AI capabilities more accessible. Key features include per-layer sensitivity analysis for mixed-precision weight allocation and LoRA fine-tuning that optimizes model performance while adhering to a specific bit budget. The integration with both OpenAI and Anthropic APIs enhances flexibility, enabling users to interact with a variety of models, including those capable of processing both text and image inputs. The significance of mlx-optiq lies in its ability to offer higher quality and efficiency in model performance through data-driven mixed-precision quantization. By allowing models to retain more quality than traditional uniform 4-bit models at similar sizes, mlx-optiq optimizes memory usage on Apple devices. The solution supports a range of models available on Hugging Face, such as the Gemma and Qwen families, with the largest model, Gemma-4, achieving a Capability Score of 79.7 while maintaining a relatively small disk footprint. This advancement in local model deployment underscores a shift towards making complex AI tools more user-friendly and efficient for individual developers and researchers.