GPT-2 implementation in Modular MAX (github.com)

Modular MAX has added a production-ready GPT-2 implementation you can run with a single command (max serve --model openai-community/gpt2 --custom-architectures ../max-gpt-2) and native GPU support. The release bundles two important inference optimizations — paged KV caching and FlashAttention — so GPT-2 variants can run faster and handle much larger contexts on modern GPUs. This is significant for researchers and deployers because it brings higher-throughput, lower-cost inference to a familiar open checkpoint while preserving the ability to swap in custom architectures. On an Nvidia RTX 5090 test, the optimizations yield dramatic gains: prompt-processing throughput rose from ~3.7K tok/s to ~30.7K tok/s (≈8×), and token-generation throughput jumped from ~14.9 tok/s to ~250.1 tok/s (≈17×). Paged KV caching reduces in-memory key/value footprint by spilling or paging KV state, enabling longer contexts without linear GPU memory growth; FlashAttention accelerates and memory-optimizes the attention kernel. Together these changes make GPT-2 inference far more practical for interactive and high-volume workloads, and the modular MAX integration means developers can apply the same pattern to custom GPT-2-derived architectures.