How to Use Decision Models for Evals (hamel.dev)

Recent advancements in decision models, such as TypeSafe’s Jev, Perplexity’s Decider, Liquid’s d1, Cloudflare’s Clef, and OpenAI’s Decisions API, are positioning them as cost-effective alternatives for automated evaluations compared to traditional large language model (LLM) judges. These models offer rapid classification capabilities, but selecting the optimal model hinges on specific cost, performance, and latency requirements. Notably, many users default to a threshold of 0.50 for classifying outputs, which can hinder performance; adjusting this threshold is crucial to maximizing precision and recall while reducing false positives and negatives. In practical applications, a recent study evaluated these models by utilizing conversation logs from an apartment leasing agent, focusing on their ability to identify errors like failed bookings. Performance metrics, such as the F1 score, were analyzed alongside API costs. OpenAI's Decisions achieved the highest F1 score, but at a significantly higher cost compared to others like Perplexity. Calibration techniques, such as Platt scaling, were suggested to refine the models' predicted probabilities, thus enhancing their reliability for pass/fail decisions. This ongoing evaluation process highlights the importance of tuning decision thresholds and understanding model performance in relation to specific business goals, encouraging a more tailored approach in utilizing decision models in AI applications.