Adrianco's Retort: measure how reliable, fast and expensive your LLM is (adrianco.medium.com)

Adrianco has introduced a new tool called Retort, designed to evaluate the reliability, speed, and cost of different versions of the Claude Code models across various programming languages. Unlike traditional benchmarks that maintain a fixed setup and overlook real-world variables, Retort applies a statistical Design of Experiments methodology to provide a nuanced assessment. The headline metric focuses on how often a model produces correct outcomes, stressing that functions must meet all requirements to be deemed successful. Initial experiments revealed that newer models yield higher reliability—with the Opus-4.8 achieving 100% correctness on difficult tasks, albeit at greater cost and slower speeds. This development is significant for the AI/ML community as it highlights the trade-offs between model performance and resource expenditure, informing decisions on model deployment based on project requirements. Retort's findings indicate that while newer models like Opus-4.8 deliver greater reliability for challenging tasks, they come with substantial increases in runtime and costs—showing about three times the expense and roughly 50% slower execution compared to older versions. Moreover, the tool underscores the importance of context, suggesting that the choice of programming language and task type can greatly influence outcomes, thus advocating for a more granular analysis over average performance metrics when selecting models for real-world applications.