Training an LLM in Swift, Part 1: Taking matrix mult from Gflop/s to Tflop/s (www.cocoawithlove.com)

In a new article series, a developer explores optimizing matrix multiplication code for training a Large Language Model (LLM) in Swift on Apple Silicon, aiming to enhance performance beyond established C implementations. The initiative is significant for the AI/ML community, as it emphasizes the potential for high-performance machine learning directly in Swift, rather than relying on higher-level frameworks. The writer's self-taught approach is set against a backdrop of slow initial performance; the basic Swift implementation runs approximately 15-20 times slower than its C counterpart. The author highlights technical hurdles, including issues with Swift's array handling and compiler optimizations, particularly the lack of fused-multiply-add operations compared to C, which greatly affects performance. With the introduction of Swift's MutableSpan and insights on compiler flags, the author is making progress toward optimizing training iterations and refining the LLM code. This work not only demonstrates the feasibility of Swift for ML tasks but also showcases the ongoing challenges developers face in achieving competitive performance in training neural networks without established machine learning libraries.