Uncensored LLMs in agent loops: 31 models, 5 tasks, most fail half the time (openuncensored.com)

A recent evaluation of 31 uncensored language models (LLMs) revealed significant reliability issues when tasked with complex multi-step operations. The study assessed the models through five specific tasks—including file editing and debugging—capping each at 14 turns. While models based on high-scoring originals (60+) completed an average of 38% of tasks, those built on weaker originals completed just 27%. Notably, the study found that up to 41% of weaker models repeatedly looped through actions without progress, indicating severe limitations in their task execution capabilities. This analysis sheds light on the critical performance challenges facing the AI/ML community, where even models derived from more intelligent originals frequently fail to produce reliable outputs. The findings suggest that simply being built on a competent original does not guarantee robust performance, as all models showed a propensity for malfunctions or task failures. The results call for enhanced testing and validation processes, particularly for uncensored LLMs, emphasizing the need for a deeper understanding of their operational boundaries and the potential pitfalls of deploying such systems in practical applications. Further trials, including more comprehensive evaluations and the introduction of native function calling, are expected to provide deeper insights into improving model reliability.