Running Kimi K3 on a M1 Mac (github.com)

A research project named Deltafin has successfully run the Kimi K3 model, which features an impressive 2.8 trillion parameters, on an M1 Mac, specifically using an M1 Max chip with 64 GB of RAM. While the performance is quite slow at approximately 16 seconds per token, the project demonstrates the feasibility of running large Mixture-of-Experts (MoE) models on consumer-grade hardware. Users can choose between a full 1.7 TB installation or a lightweight 215 GB streaming option, facilitating experimentation without substantial local storage commitments. The model operates by selecting from a pool of 16 experts across 92 layers, significantly limiting the amount of data processed per token, making local inference possible despite the model's size. This development is significant for the AI/ML community as it showcases the potential of deploying advanced models in everyday settings, particularly on macOS systems. The experiment highlights the impact of optimizing RAM usage, memory bandwidth, and computational efficiency in large-scale models. Although the current implementation is not suitable for interactive applications, it sets a precedent for future research into streaming inference techniques and resource-efficient model architectures, ultimately pushing the boundaries of accessibility for machine learning applications.