Show HN: OMLX – MLX inference server with paged SSD KV caching for Apple Silicon (github.com)

OMLX has launched a new inference server optimized for macOS, specifically tailored for Apple Silicon devices (M1/M2/M3/M4). This innovative tool simplifies LLM (Large Language Model) usage by enabling continuous batching and utilizing SSD storage for key-value (KV) caching. Users can manage everyday models directly from the macOS menu bar, allowing them to pin frequently used models, automatically swap larger models on demand, and maintain context across conversations seamlessly. The server supports various LLMs and includes built-in optimization for Claude Code, further enhancing productivity for coding tasks. The significance of OMLX lies in its ability to balance convenience with control, a long-desired feature within the AI/ML community. By integrating features such as paged SSD caching, multi-model serving, and a user-friendly web admin dashboard, OMLX improves local LLM implementation for real-world applications. Additionally, it offers compatibility with OpenAI and Anthropic APIs, allowing developers to easily connect through familiar endpoints. The architecture is built on FastAPI, supporting efficient model management and monitoring, paving the way for more efficient deployment of machine learning models on Apple hardware.