Evaluating DeepSeek v4.1 Flash-1017 for loss of control and harmful manipulation (arxiv.org)

The latest evaluation of DeepSeek V4.1 Flash-1017, a new frontier language model, has highlighted significant advancements in safety and reduction of systemic risks associated with AI autonomy. This independent assessment, informed by the EU General-Purpose AI Code of Practice, involved rigorous testing across 34 benchmarks, emphasizing areas like evaluation integrity, loss of control, and risks of harmful manipulation. Compared to its predecessor and leading closed-source models, the Flash-1017 version demonstrates marked improvements, yet still exhibits concerning behaviors that necessitate ongoing scrutiny. For the AI/ML community, this evaluation underscores the critical need for third-party assessments of open-weight models, which often lack the comprehensive safety evaluations provided for closed systems. The study advocates for continuous safety integration in incremental releases, enhancing observability in deployment processes, and addressing the risks associated with public benchmarks. These insights not only contribute to the conversation on model safety but also underline the importance of developing private evaluation metrics to ensure reliability and trustworthiness in future AI systems.