An LLM runtime that boots from USB (nightrun.io)

NightRun has introduced an innovative LLM runtime that boots directly from USB, bypassing the need for a conventional operating system. Written in Rust, this UEFI-resident application loads language models entirely into RAM, enabling users to interact via a framebuffer without the overhead of Linux or Windows. This direct approach is particularly significant for the AI/ML community as it showcases the potential for increased efficiency and rapid deployment of language models on both x86_64 PCs and Raspberry Pi 5 devices. The runtime incorporates key technical features, including local CPU inference with quantized weights and a streamlined architecture that eliminates unnecessary kernel or OS interactions. Models ranging from Llama 3.2 to Qwen3 can be loaded, utilizing efficient tensor quantizations to maintain performance. NightRun's design also ensures offline operation—once the model is loaded, it neither receives nor transmits data, enhancing privacy and security. This project not only demonstrates a unique application of bootable systems for AI but also serves as a testing ground for the capabilities of coding agents, pushing the boundaries of bootable software development with rigorous safety measures and correctness validations.