On-device Qwen benchmark: the 4B ties the 9B, and more findings (github.com)

A new benchmark reveals surprising insights about the performance of Qwen’s 4 billion (4B) and 9 billion (9B) parameter models, run entirely on an Apple M5 Max. The study highlights that the smaller 4B model often ties with or outperforms the more extensive 9B model on tough math-reasoning challenges, a finding that suggests diminishing returns on scaling model size without correspondingly enhanced performance. A critical factor is identified as decoding temperature, where switching to a greedy, deterministic decoding method significantly improved the performance of all models, showing that previous evaluations may have misrepresented their capabilities due to noise. The 35 billion parameter MoE (Mixture of Experts) model, while being the largest, operates fewer active parameters per token and excels in solving the hardest problems, demonstrating the importance of efficient capacity distribution and extensive training. The implications for the AI/ML community are profound, indicating that model size alone does not determine performance, and revealing the need for careful consideration of evaluation methods and settings when assessing model capabilities. This benchmark method is accessible and can be utilized across various platforms, encouraging reproducibility and broader testing in the field.