From 75% to 99.6%: The Math of LLM Ensembles (www.shibaprasadb.com)

A recent project demonstrated a novel approach to improving the accuracy of large language model (LLM) API calls by leveraging ensemble techniques, boosting accuracy from 75% to an impressive 99.6%. Initially struggling with production-level accuracy while counting elements from a list, the author identified a consistent undercounting bias in the model's responses. By applying the "wisdom of crowds" principle—similar to the Random Forest algorithm—the author aggregated results from multiple API calls using a max function to filter out undercounts, allowing for a more reliable output with minimal overcounting issues. This method marks a significant advancement for the AI/ML community, showcasing how a deeper understanding of a model's failure modes can lead to smarter utilization strategies rather than merely seeking model enhancements. With evaluations indicating that three API calls offer 98.4% accuracy, utilizing four calls achieved nearly perfect results at a cost multiplier of four times. This approach emphasizes that enhancing API efficiency in production settings can offset costs while achieving desired accuracy, highlighting the importance of both understanding and strategically addressing the inherent biases of LLMs in practical applications.