Claude was faster and cheaper. We took the scenic route anyway (blog.grainulation.org)

A recent study introduces a novel approach called "Verify, Then Vote," aimed at enhancing the accuracy of responses from multiple AI models. Instead of relying solely on majority voting—which can reinforce shared inaccuracies among models—the authors implemented a systematic method to evaluate the sources cited by each model. By utilizing a designated verification model, they ensured that the final answer was based not just on votes but confirmed support from credible sources. Tested on five local models run on an Apple M3 Pro, the method yielded a significant accuracy increase from 77.1% to 85.0%. This development is significant for the AI/ML community as it addresses the challenges of model discrepancies while managing computational resources more effectively. Notably, the research demonstrated that the new rule performed well even with a fixed limit on wrong answers, enhancing robustness in decision-making. By enabling the rerunning of decisions without additional model calls and leveraging saved checks, the approach proves promising for future applications. The accompanying GitHub repository offers access to the paper and code, inviting further exploration of the methodology and its potential refinements.