CUDA+ROCm dual-back ends LLM inference for mixed-vendor GPUs (up to 48% faster) (github.com)

A new development in AI/ML has enabled mixed-vendor GPU support for Large Language Model (LLM) inference through a dual-backend approach using NVIDIA's CUDA and AMD's ROCm, resulting in performance improvements of up to 48%. This innovation allows primary model execution with the CUDA framework while utilizing the HIP module for complementary gradient-based linear model (GLM) experts. This dual architecture is designed for systems equipped with both NVIDIA and AMD GPUs and maintains efficient allocations and runtime management via a version-controlled C ABI, demonstrating a significant leap in flexibility for AI developers who need to leverage hardware from different vendors. The technical implications are profound, as the integration supports configurations with enhanced persistent expert storage, improved memory handling, and a streamlined build process that caters to both Linux and Windows environments. With benchmark results showcasing substantial throughput increases—measured at 45.10 to 69.05 tokens per second in various configurations—the framework is not only an effective solution for performance but also offers a clean and modular plug-in system for future enhancements. By ensuring device namespace separation and providing robust validation through extensive testing, this update underscores a pivotal shift towards interoperability in GPU computing, empowering researchers and developers to maximize the performance of their AI applications regardless of the GPU vendor.