LLMs run on top of an OS designed for code, not weights (github.com)

A groundbreaking approach to optimizing large language models (LLMs) on memory-constrained hardware was announced, leveraging a novel weight paging technique that treats model weights similarly to how operating systems manage virtual memory. This method allows models with over 200 billion parameters to run efficiently on machines with only 16 GB of RAM by prefetching weight blocks needed for inference, significantly improving performance and reducing energy consumption during operations. The significance of this development lies in its potential to make sophisticated LLMs accessible on devices that traditionally could not support them due to hardware limitations. By employing a predictive prefetching mechanism based on Markov chain models trained from actual inference data, this approach anticipates which weight blocks will be required next, hiding SSD read latencies and mitigating stalls during token generation. Benchmarks showed a 1.16x improvement in throughput compared to traditional methods, alongside a notable reduction in I/O operations and energy usage, making this solution particularly suitable for devices like Apple Silicon, which can benefit from its high bandwidth and unified memory architecture.