An inference server for AMD GPUs, in C++ and HIP (codeberg.org)

A new inference server called Radiance has been released, designed specifically for AMD GPUs and built using C++ and HIP. This server supports model architectures, kernel libraries, and quantizers as dynamic shared objects (.so) plugins, and it is compatible with an OpenAI-style HTTP API. It targets AMD's RDNA4 architecture and allows for smooth deployment of large language models (LLMs) like Qwen3.8 and MiniCPM5 with different precision levels (FP8, bf16) through Docker containers, making model management and scaling more efficient. This development is significant for the AI/ML community as it provides a robust solution for deploying models on AMD hardware, which has seen less focus compared to NVIDIA platforms. The server can start on any AMD GPU, with specifics outlined for Radeon AI PRO R9700 cards, enhancing accessibility for developers and researchers who prefer or require AMD's architecture. Additionally, the open-source nature and detailed documentation facilitate collaborative improvements and innovations, potentially broadening the landscape for LLM deployment and performance optimization across various applications.