Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090 (www.gilesthomas.com)

Recent benchmarking of the Qwen 3.6 35B Mixture of Experts (MoE) model on an RTX 3090 has revealed performance insights critical for the AI and machine learning community. Utilizing a quantized 4-bit version of the model, tests indicated the model could generate tokens at around 120 tokens per second while processing prompts at nearly 2,800 tokens per second. However, the limited VRAM of the RTX 3090 constrained the effective context window to about 50,000 tokens, much lower than the model's full capability of 262,144 tokens. By offloading certain layers to the CPU, users could reclaim memory, but this resulted in significantly slower performance. The implications of these findings underscore the crucial relationship between model architecture and hardware capabilities, especially in relation to VRAM requirements for large models. The Qwen model's architecture, which activates only 3 billion of its 35 billion parameters for inference, presents unique challenges regarding memory usage and efficiency. Additionally, insights into the nuances of quantization reveal that it's a complex process akin to lossy compression, affecting how models can be run on hardware with differing capabilities. This benchmarking effort not only provides practical guidance for users hoping to deploy large, multimodal models but also sparks discussions about the next generation of hardware solutions, such as the upcoming RTX 5090, which promises significantly improved VRAM and processing power.