Janas: Run LLMs larger than RAM by streaming experts from SSD (github.com)

Janas-LLM, a newly released component of the Janas suite, enables the execution of large language models (LLMs) exceeding the physical RAM capacity of standard laptops by streaming model parameters directly from an NVMe SSD. Demonstrated on a laptop with 32 GB of memory, Janas-LLM achieves impressive speeds, outperforming traditional frameworks like llama.cpp across various 48 billion parameter models. For instance, it reads prompts at rates up to 62 tokens per second, which is significantly higher than llama.cpp’s 16 tokens per second for the same model, showcasing the efficiency of streaming expert weights while caching frequently used experts in RAM. This development is significant for the AI/ML community as it democratizes access to high-performance LLMs by allowing broader usage on everyday hardware, regardless of GPU capabilities. Since Janas-LLM operates independently of large memory requirements, it could facilitate research and application in environments with limited resources. The implications for model deployment are profound; researchers and developers can now leverage powerful LLMs without investing in expensive, high-RAM systems, potentially accelerating innovation in natural language processing and AI-driven applications.