Atlas: An LLM inference engine written from scratch in Rust and CUDA (atlasinference.io)

A new inference engine for large language models (LLMs) called Atlas has been introduced, built from the ground up using Rust and CUDA without relying on PyTorch or Python. This lean solution offers a ~2.5 GB binary, boasting a significant performance boost—up to three times faster than current dominant frameworks like vLLM, which carries 20+ GB of dependencies. Atlas enhances efficiency through direct compilation techniques that eliminate interpreter overhead, including a hand-tuned selection of CUDA kernels tailored to different model architectures. The engine supports multi-token prediction, enabling it to generate multiple tokens in a single forward pass, thus increasing throughput dramatically. The significance of Atlas lies in its optimization for specific hardware configurations, such as Nvidia's DGX Spark and future support for RTX 6000 Pro Blackwell chips, indicating a shift towards community-driven enhancement based on user feedback. With real-world tests showing substantial speed improvements on various models, Atlas aims to provide a streamlined and effective option for developers and researchers in the AI/ML community. The project prioritizes a customizable and efficient experience by offering OpenAI-compatible API integrations and support for multiple modalities, laying the groundwork for future expansions and enhancements.