Fractal-BLT – Zero-allocation .NET 10 MoE runtime streaming NVMe to GPU (github.com)

Fractal-BLT has launched a groundbreaking runtime for artificial intelligence that leverages .NET 10 NativeAOT to facilitate zero-allocation streaming of mixture-of-experts weights from NVMe storage directly to GPU memory. This innovation bypasses the limitations of traditional AI frameworks—which often rely heavily on memory management strategies that can slow down performance—by avoiding managed heap allocations and reducing latency with unbuffered direct I/O. Key technical components include a powerful FractalGNN router, which performs efficient graph message passing, and direct memory access (DMA) for seamless data transfer between NVMe and VRAM. The significance of Fractal-BLT lies in its potential to revolutionize local AI inference applications by drastically minimizing overhead caused by software stacks, achieving a startup memory footprint of approximately 10 MB compared to 1.8-3.5 GB for standard Python/PyTorch setups. With its ability to handle "infinite" VRAM model capacities and impressive metrics for byte patching and message routing, Fractal-BLT positions itself as a formidable tool for AI developers seeking high-throughput, low-latency solutions. This open-source project not only enhances the efficiency of AI workloads but also promises the flexibility and performance needed for advanced applications in both enterprise and mobile environments.