Benchmarks show scores. Dashboards show usage. What ships? (pragmatikos.ai)

OpenCode 2.0 has launched a new scoring system, Pragmatikos, aimed at evaluating AI model pairings based on real work outcomes rather than controlled benchmarks. While traditional benchmarks assess the capabilities of individual models in isolation, Pragmatikos focuses on the effectiveness of combined model use in actual coding sessions. This system ranks various model pairings by analyzing the number of successful commits, the effort required, and the quality of the resulting code, providing a more pragmatic view of productivity in software development. This development is significant for the AI/ML community as it addresses a crucial gap in understanding how different AI models collaborate in real-world scenarios. The rankings are derived from 1,210 cycles and reflect actual developer experiences, allowing for insight into which combinations yield the best results. However, the system also acknowledges its limitations, such as potential biases from a small data pool and the variance in developer skill. As the pool of submissions grows, the hope is to refine this method, enabling a better understanding of model interactions and improving decision-making for developers choosing AI tools.