Trying VLLM Ideas on Apple Silicon with MLX (WIP) (github.com)

A new project, vllm-mlx, is underway to enable GPU-accelerated Large Language Model (LLM) inference on Apple Silicon Macs, including the M1, M2, M3, and M4 processors. This initiative integrates Apple's MLX framework, which features unified memory and Metal kernels, ensuring efficient inference with capabilities like 4-bit quantization for running large models on constrained memory. The development includes support for both language and vision-language models, allowing users to handle multimodal input, such as text, images, and videos, through an OpenAI-compatible API. This project is significant for the AI/ML community as it provides an accessible way for developers to leverage powerful AI capabilities on consumer-grade hardware, optimizing performance without the need for complex data transfer processes. The early phases have successfully implemented basic LLM functions like text generation, streaming, and a chat interface, while future enhancements will bring full multimodal support, including image and video analysis. With a focus on performance optimizations like request batching and KV cache optimization, vllm-mlx promises to streamline the use of advanced AI models on Apple devices, potentially expanding the user base and applications for AI technologies.