MDMA: An open-source LLM inference engine built with Rust (github.com)

MDMA, an open-source language model (LLM) inference engine crafted in Rust, has been unveiled to enable individuals to run language models on their existing hardware. This engine, which includes vital components such as a GGUF loader, tokenizers, and transformer decoders, is designed for high-performance inference by allowing various machines, from desktops to servers, to collaborate in a private network without compromising data security. Users can utilize their devices more effectively, potentially achieving impressive throughput rates of 1,000 tokens per second on suitable hardware. The significance of MDMA for the AI/ML community lies in its focus on decentralized computing and the prioritization of user privacy. Unlike traditional cloud-based models, MDMA eliminates reliance on external servers, employing mTLS for secure communication within a self-managed federation of machines. Its architecture includes multiple backends supported by different computing platforms—such as CUDA for NVIDIA GPUs, Metal for Apple devices, and Vulkan for AMD—ensuring broad accessibility. The unique memory allocator upfront checks memory availability, enhancing stability and performance by preventing runtime failures due to memory exhaustion. MDMA's experimental phase invites community feedback and contributions, fostering an inclusive approach to refining the technology.