JustFit: Run Qwen 27B Fast with its full 256K context on a 24 GB Mac (github.com)

JustFit has launched a powerful local inference system capable of efficiently serving a 27 billion parameter language-and-vision model on Macs with memory constraints. This innovative solution leverages a compressed context and an intelligent management of CPU, GPU, and Apple Neural Engine (ANE) resources, allowing users to handle long histories and multiple requests concurrently. With warm-prefix tests reaching an impressive aggregate page capacity of 320K across two live requests, JustFit demonstrates significant advancements in memory utilization and processing speed, with version 2 boasting a median decode rate 2 to 2.5 times faster than its predecessor. This development is highly significant for the AI/ML community as it enables powerful machine learning applications on consumer-grade hardware, making advanced AI capabilities more accessible. JustFit's efficient architecture optimizes AI processing by integrating various hardware components while supporting OpenAI-compatible APIs for a seamless user experience. Additionally, the system operates effectively under macOS 27.0+, ensuring compatibility with the latest Apple silicon, thus fostering further innovation and research in the field of local AI deployments.