Show HN: NightRun, bare metal LLM inference, no OS, boots from USB (github.com)

NightRun, a groundbreaking local LLM runtime, has been announced, designed to boot from a USB drive and operate without a conventional operating system. Instead of relying on a traditional OS, NightRun initializes directly from a machine's firmware, loading a quantized model into RAM and providing users with a terminal for interaction. This innovative approach allows for reduced loading times and on-device performance, running models like Llama 3.2 directly on x86_64 PCs and Raspberry Pi 5 without any background system processes, thus streamlining the user experience. This development is significant for the AI/ML community as it highlights a novel method of deploying large language models that prioritizes simplicity and performance. Written in Rust, NightRun employs sophisticated techniques such as batching token processing and strict verification of model integrity, ensuring efficiency and correctness throughout the entire generation process. With capabilities for various model families and rigorous testing against established implementations, NightRun aims to offer a robust and secure platform for offline LLM inference. Its unique architecture could inspire future advancements in bare-metal AI deployments, emphasizing minimalism in design while maximizing performance and user accessibility.