OpenAI has a LOT of work to do if they think Luna can compete with Jev (anth.us)

OpenAI recently introduced its Decisions API, built on a version of its new model, GPT-6 Luna, during its DevDay event. This API aims to facilitate more seamless decision-making processes in software by selecting answers from predefined options with a significant reliance on model confidence. However, initial evaluations raise concerns about Luna’s accuracy, particularly when it claimed high confidence (over 99%) but achieved correct answers only 68% of the time across 3,600 reasoning tasks. The results highlight a stark contrast with an existing model, Jev, which maintained 98.9% accuracy under similar conditions. This situation is significant for the AI/ML community as it underscores the ongoing challenges of model calibration and reliability in decision-making contexts. Luna's overconfidence suggests that without proper adjustments, the Decisions API may lead to misguided outcomes, potentially invalidating its utility for critical applications. As the technology evolves, the expectation is for OpenAI to address these calibration issues, ideally transforming confidence reporting into a robust, transparent aspect of future model development. The community is left hoping for a revised approach to model confidence, ensuring it can be trusted for high-stakes applications.