Fleet: Hierarchical Task-Based Abstraction for Megakernels on Multi-Die GPUs (arxiv.org)

A significant advancement in GPU programming has been presented with the introduction of Fleet, a hierarchical task-based abstraction designed for megakernels on multi-die GPUs. Traditional programming models like CUDA and HIP lack the ability to leverage the chiplet-level architecture that modern GPUs employ, resulting in inefficient memory usage and increased latency in memory-bound tasks, particularly in applications such as large language model (LLM) inference. Fleet addresses this gap by introducing Chiplet-tasks, which create a new level of hierarchy that allows tasks to be organized around individual chiplet caches. This innovative approach facilitates better cache utilization and reduces memory traffic by enabling tasks to share resources through a chiplet's L2 cache. Fleet has shown promising results when tested on the AMD Instinct MI350 with the Qwen3-8B model; it achieved a 1.3-1.5x reduction in decode latency at smaller batch sizes compared to the benchmark vLLM. The implementation of per-chiplet scheduling and cooperative weight tiling led to a significant increase in L2 cache hit rates and a reduction in high-bandwidth memory (HBM) traffic by up to 37% at larger batch sizes. These improvements highlight the potential for Fleet to revolutionize performance in memory-intensive workloads by optimizing resource management at a granular level, providing a substantial speedup over conventional chiplet-unaware models.