A Benchmarking Framework for Software-Based GPU Virtualization Systems (arxiv.org)

A new benchmarking framework named GPU-Virt-Bench has been introduced to address the growing demand for efficient GPU resource sharing in AI-driven workloads, particularly in cloud and container environments. While NVIDIA's Multi-Instance GPU (MIG) technology offers hardware-level isolation, it is not universally applicable due to its limitation to high-end datacenter GPUs. The GPU-Virt-Bench framework evaluates software-based GPU virtualization systems such as HAMi-core and BUD-FCSP using 56 performance metrics organized into 10 key categories, including overhead, isolation quality, and LLM-specific performance. This framework is significant for the AI and ML community as it enables a systematic comparison of software virtualization solutions against the ideal MIG performance, offering invaluable insights for optimizing GPU resource allocation in multi-tenant setups. By assessing critical performance characteristics, GPU-Virt-Bench aids practitioners in making informed decisions regarding production deployments. The results from the framework highlight the capabilities and limitations of various software virtualization approaches, positioning it as a vital tool as GPU-accelerated applications continue to proliferate.