Gemma-4-31B at 256K context on a $1,400 AMD GPU – measured, with patches (github.com)

The recent successful integration of the Gemma-4-31B model with TurboQuant and HIP graphs on AMD's RDNA4 architecture marks a significant milestone in AI/ML for long-context processing. Operating on a $1,400 AMD Radeon AI PRO R9700 GPU, the model achieved a notable 735 tokens per second prefill rate and maintained crash-free decoding at a full 256,000 token context. This achievement not only showcases the viability of running large models on AMD hardware but also addresses prior concerns regarding VRAM limitations, highlighting that configuration missteps were the primary culprits behind earlier performance issues. The integration hinged on two critical patches that allowed TurboQuant's quantized key-value cache to work in tandem with HIP graphs, optimizing both prefill and decode speeds. The findings also revealed several configuration traps that could degrade performance by five to ten times if left unaddressed. Importantly, this work opens doors for more efficient long-context inference applications, such as in agentic coding environments, thereby enriching the AI toolkit for developers while pushing the boundaries of what is feasible with existing GPU infrastructure.