Show HN: Modeljury – a jury of cheap LLMs that escalates the hard cases (github.com)

Modeljury, a new AI tool presented on HN, utilizes a collaborative approach by employing a panel of five lightweight language models to tackle complex decision-making tasks. The tool operates by having models vote on decisions; if they agree, the answer goes through, while disagreements prompt a human review. Tested on over 2,800 decisions from various datasets, Modeljury successfully ships 71% of decisions without human input, with a commendable 89.6% accuracy rate. This efficiency comes at a fraction of the cost compared to more complex models like Claude Opus, which, while more accurate overall, is ten times pricier. The significance of Modeljury lies in its innovative use of multiple models to mitigate the risk of errors and biases that come from relying on a single AI. Each dissenting model provides explanations for their choices, enhancing transparency for human reviewers. This approach minimizes the risk of a single point of failure and is particularly effective in flagging controversial decisions for further examination. With ease of implementation via a Python library, Modeljury not only advances the practical application of AI in decision-making processes but also sets a precedent for future developments in collective AI systems.