Local AI app that runs an 80B model on an 8GB Mac (yanahchat.com)

A groundbreaking local AI application, Yanah, has demonstrated the capability to run an 80-billion-parameter model on an 8 GB MacBook, defying previous assumptions that such a feat was impossible. Traditional models, like stock mlx-lm or llama.cpp, face severe limitations due to their large memory footprints and quantization issues, but Yanah employs a unique approach by using a hybrid Mixture-of-Experts architecture. This model allows it to activate only a fraction of its parameters at a time, maintaining a minimal memory footprint while ensuring high performance in token processing, achieving speeds of 5 to 8 tokens per second under constrained conditions. Significantly, Yanah's breakthroughs stem from surgical three-tier quantization and zero-copy streaming, which drastically reduces memory requirements and enhances processing efficiency. By leveraging Apple Silicon’s NVMe SSD for expert streaming and employing sophisticated caching mechanisms, Yanah can keep essential components in RAM while dynamically loading experts as needed. This innovative architecture not only preserves approximately 90-92% of the original model's intelligence but also allows seamless multi-turn conversations and state management. Yanah's approach could reshape local AI application development, making powerful models accessible on consumer-grade hardware, which is a major advancement for the AI/ML community.