Modern GPU Programming for MLSys (mlc.ai)

A new book titled "Modern GPU Programming for MLSys" has been released, focusing on optimizing GPU kernels crucial for machine learning workloads. As machine learning systems increasingly rely on sophisticated GPU architectures to enhance performance, this resource aims to equip developers with a deep understanding of both the hardware and the high-level programming models necessary for building efficient kernels. Key topics include the development of advanced kernels like fast matrix multiplication (GEMM) and FlashAttention, emphasizing techniques such as data layout, asynchronous memory operations, and coordination. The book is structured to first build a fundamental understanding of GPU architecture before progressing to practical applications using the TIRx Python domain-specific language (DSL). This approach allows users to engage with low-level control while learning through hands-on examples. The content stems from the Machine Learning Systems course at Carnegie Mellon University, aiming to demystify the complexities of modern GPU programming and empower AI/ML practitioners to create state-of-the-art performance optimizations. With a focus on the Blackwell generation of GPUs, this resource is poised to be significant for advancing the capabilities of machine learning systems.