Does the code compile? Five local models on a DGX Spark, five runs each (borisivanov.com)

A recent benchmarking study evaluated five AI models using a DGX Spark setup, focusing on whether their code compiles and tests pass rather than just throughput performance. This approach, leveraging tools like Ollama and llm-compile-bench, aims to help developers understand the practical usability of these models in coding scenarios. The models tested included Qwen3.8-27B, gpt-oss-120b, and DeepSeek V4 Flash, among others, with results revealing significant variances in reliability and speed. The findings indicate that while Qwen3.8-27B emerged as the most reliable option, gpt-oss-120b is faster for producing passing answers. Notably, the models struggled particularly with concurrency tasks, suggesting critical weaknesses in handling complex coding scenarios. The study emphasizes that outright speed in token generation doesn't equate to overall effectiveness—citing that models with lower token throughput, like Qwen3.8-27B and gpt-oss-120b, provided more consistent and usable outputs. This nuanced approach offers valuable insights for developers in selecting the right model for their specific AI application needs, highlighting the importance of comprehensive testing beyond simple performance metrics.