An 8.6 GB model that serves only 7 requests a second (mapathak-commits.github.io)

A recent experiment by Manas Pathak highlights the surprising limitations of self-hosting language models, specifically focusing on the Qwen3.5-4B model. Using an NVIDIA A10G GPU, which has 23 GB of memory, Pathak discovered that despite the model's weights occupying only 8.6 GB, the server could only handle about seven requests per second under load. This finding challenges the common belief that simply fitting a model into GPU memory guarantees high throughput, revealing that the primary bottleneck lies in the decoding process rather than memory constraints. This research is significant for the AI/ML community as it emphasizes the complexity of workload management in deploying language models. While the hybrid-attention design of Qwen3.5-4B allows for efficient KV memory utilization, the model's reliance on heavy weight-streaming during decoding limits request throughput. The findings encourage developers and engineers to rethink performance metrics, focusing on which parameters—be it memory, model size, or decoding time—will impact service capacity most significantly, rather than assuming that memory availability alone dictates performance.