Build your own decision model (nishtahir.com)

A recent article introduces the concept of "system one" decision models, highlighting a new approach to generating responses from language models using constrained output techniques. By limiting the model's output to fixed options (A, B, C, D, E), this method simplifies the decision-making process, allowing the model to make predictions in a single pass rather than multiple iterations. For instance, using the Qwen/Qwen3-1.7B model, the authors demonstrate its efficacy in answering questions with precise outputs based on the highest probability selection from pre-defined answers. Initial tests showed the model could achieve a 59.38% accuracy rate on the CommonsenseQA dataset, with performance improving slightly after fine-tuning. The significance of this work lies in addressing the reliability of AI models’ confidence scores, which often do not reflect actual accuracy. The authors point out that many models display overconfidence in their predictions, leading to mismatched confidence and correctness. To remedy this, they propose calibrating output probabilities through temperature scaling, enhancing the model's accuracy. This advancement not only benefits practical applications by ensuring more reliable AI interactions but also provides a framework for others to experiment with and improve their own models, as resources have been shared on GitHub.