Tested 18 new Local LLMs on two RTX 5060 Ti cards: none replaced Qwen3.8-27B (siimtolk.github.io)

A recent experiment evaluated eighteen new local language models (LLMs) to determine if any could successfully replace the existing personal assistant model, Qwen3.8-27B. The testing pipeline, run overnight, assessed each model's performance across numerous tasks. Despite the promising capabilities of some contenders, none could surpass Qwen3.8-27B in terms of reliability, speed, or adaptability to specific requests, particularly in languages with complex grammatical structures like Estonian. This evaluation is significant for the AI/ML community as it highlights the limitations of relying solely on public leaderboard performance when selecting models for practical applications. The testing framework utilized a customized suite of 97 real-world scenarios, allowing for a nuanced understanding of model capabilities that aggregate scores alone cannot capture. Key technical insights include the efficient use of two NVIDIA RTX 5060 Ti cards, the innovative approach of continuous overnight testing, and a focus on maintaining production stability while rigorously evaluating new LLMs, underscoring the necessity for tailored benchmark assessments in the ongoing evolution of AI technologies.