Popular Attention Alternatives: GQA, MLA, SWA (sebastianraschka.com)

The repo/book adds a focused, hands‑on treatment of three popular attention alternatives—Grouped‑Query Attention (GQA), Multi‑Head Latent Attention (MLA), and Sliding Window Attention (SWA)—showing how they replace standard Multi‑Head Attention (MHA) in modern LLMs. GQA reduces compute and memory by grouping queries so many heads share keys/values (used in Llama 4, gpt‑oss, Qwen3, Gemma 3), MLA introduces a small set of latent tokens that compress the sequence into a lower‑dimensional attention bottleneck (DeepSeek V3), and SWA applies overlapping local windows to get near‑linear scaling for very long contexts (used by Gemma 3 and others). Each approach trades full global expressivity for efficiency, so the repo explains architecture changes, complexity reductions, and practical hyperparameter choices. Why it matters: these patterns are how current large models scale to longer contexts and larger heads without blowing up memory or flops, and they directly affect pretraining, fine‑tuning, and inference latency. The materials include code examples, performance analysis, KV‑cache for faster generation, and guides for converting GPT implementations to Llama‑style models, making it easy to experiment with and benchmark these attention variants in real training and instruction‑finetuning workflows.