Show HN: Mlx-serve – LLM inference server for Apple Silicon, written in Zig (mlxserve.com)

A new open-source application called Mlx-serve has been launched, enabling users to run large language model (LLM) inference directly on Apple Silicon Macs. This lightweight server, developed in Zig, allows users to engage in various AI activities—such as chat, code generation, image manipulation, and voice tasks—entirely offline, ensuring data privacy as it never leaves the machine. The application boasts enhanced performance, outperforming existing solutions like LM Studio by 35%, and supports a wide range of models and tasks from basic chat responses to complex image and video editing. Mlx-serve introduces several innovative features, such as speculative decoding to expedite outputs, built-in memory tools, and a unique menu-bar application for seamless interaction. The server supports multiple concurrent conversations with verified accuracy under load and requires minimal setup. By allowing applications originally designed for cloud use to connect locally without modifications, Mlx-serve simplifies the user experience while maximizing efficiency—making advanced AI capabilities readily accessible on personal devices. This tool is particularly significant for the AI/ML community as it champions local computing power, data security, and enhances creative possibilities across diverse media formats, all on a user-friendly platform.