Porting a Scratch-Built 500M LLM Training Pipeline to ROCm on Strix Halo (github.com)

Recent developments have led to the successful porting of a 500M-parameter Large Language Model (LLM) training pipeline to ROCm, specifically for the AMD Strix Halo APU, while remaining compatible with other ROCm-supported hardware. This open-source initiative, initially discovered through a Reddit post, features a comprehensive end-to-end architecture encompassing data preparation, training, and fine-tuning. The model is designed using modern techniques including Grouped-Query Attention (GQA) and SwiGLU, which enhance performance and efficiency in both training and inference, distinguishing it from traditional models. This announcement holds significant implications for the AI/ML community as it demonstrates ROCm's viability as a competitor to CUDA in PyTorch environments, with minimal code alterations required for successful execution. However, it also highlights the challenges of long training times—approximately three weeks for a model of this size—underscoring the current limitations without further optimizations. The release of Plasma 1.1, which supports multi-turn conversations, is a move towards more sophisticated AI interactions, showcasing advancements in both model architecture and training techniques tailored for real-world applications.