Show HN: Run a 120B-parameter MoE on Android mid-range phone CPU-only llama.cpp (github.com)

A groundbreaking demonstration has showcased the ability to run a 120 billion-parameter Mixture-of-Experts (MoE) model, specifically the gpt-oss-120b, on a standard Android mid-range phone using only its CPU. This is significant for the AI/ML community as it indicates a major leap in making advanced machine learning models accessible on edge devices that typically suffer from limited RAM, in this case, a 12 GB phone handling a 60 GB model with astonishing efficiency. The setup achieved a generation speed of 1.3 tokens per second (tok/s) using the model's standard settings, which is dramatically faster than the 0.09 tok/s benchmark when loaded normally. The innovative technique, termed BigMoeOnEdge, allows the model to retrieve only the necessary "expert" components from flash storage as they are requested, bypassing RAM constraints. Key features like efficient expert caching, direct flash read operations, and a novel dense-weight handling policy collectively enable this impressive functionality. This system maintains full computational fidelity, providing outputs that are byte-for-byte identical to those generated entirely from RAM. The advancements in model streaming on common hardware not only enhance the versatility of mobile AI applications but also pave the way for future developments in resource-constrained environments.