Show HN: Sovereign-NB: Sub-microsecond bare-metal neural engine in Rust (UEFI) (github.com)

The Sovereign Neural Box (Sovereign-NB) has been announced as a cutting-edge, bare-metal x86-64 inference appliance that operates as a UEFI application. This compact device is designed for sub-microsecond inference and utilizes a packed ternary model to serve fixed-size input frames via UART streaming. Notably, it combines dual-inference architectures: a feed-forward multi-layer perceptron (MLP) and a recurrent causal linear attention model, allowing for efficient processing while maintaining simplicity in resource usage. The significance of Sovereign-NB lies in its no-heap design, which eliminates dynamic memory allocation—vital for real-time applications in AI/ML. By leveraging fixed-size buffers for model and data storage, the appliance caters to environments where reliability and speed are paramount. Technical details such as a lightweight file I/O system, model updates through easily replaceable weight files, and control features for attention state make it user-friendly for developers. The dual-volume structure facilitates seamless model handling and testing, emphasizing the potential for rapid deployment in diverse AI applications and innovations in edge computing frameworks.