ZML/LLMD alpha – cross platform LLM server (zml.ai)

ZML has launched the alpha version of ZML/LLMD, a versatile inference server capable of running popular LLMs like LLaMa and Mistral across five different architectures: NVIDIA CUDA, AMD ROCm, Google TPU, Intel oneAPI, and Apple Metal. This self-contained server incorporates advanced features such as continuous batching, tensor parallel sharding, and prefix caching, significantly optimizing performance. Notably, it enables zero-copy model loading directly from major platforms like HuggingFace and S3, facilitating quicker deployment without the need for manual downloads. This release is significant for the AI/ML community as it streamlines model serving across varied hardware platforms with minimal configuration required, enhancing accessibility for developers. One standout feature is the DFlash speculative decoding algorithm, which can boost throughput by up to 10x for compatible models. The system is built on ZML’s own framework, ensuring predictable latencies and efficient resource utilization. ZML/LLMD also features optimized Docker images, allowing rapid downloads and deployment, thereby reducing barriers to experimentation and implementation in AI applications.