DMA-Latte: Expanding the Reach of DMA Offloads to Latency-Bound ML Communication (arxiv.org)

Researchers have introduced DMA-Latte, a groundbreaking approach that extends Direct Memory Access (DMA) offloads to latency-sensitive machine learning communications, a capability previously limited to bandwidth-heavy scenarios. This innovation uses advanced features of AMD Instinct GPUs, allowing DMA offloads to effectively handle smaller data transfers ranging from kilobytes to low megabytes. By optimizing key ML collective operations such as all-gather and all-to-all, DMA-Latte has achieved remarkable performance improvements, offering speedups of up to 4.5 times in latency-bound environments while also reducing power consumption by 3-10% compared to existing solutions. The significance of DMA-Latte lies in its potential to enhance the efficiency of large-scale machine learning workloads, particularly during inference phases of large language models. Demonstrably, DMA-Latte can deliver up to 1.65 times lower latency and improve throughput by 1.9 times over the state-of-the-art vLLM inference framework. By unlocking the full capabilities of AMD Instinct GPU runtime innovations, this breakthrough not only narrows the performance gap for efficient communication in latency-bound scenarios but also sets a new standard for optimizing GPU resources in AI applications.