AgentMafia – A Social Deduction Benchmark (twitter.com)

A groundbreaking benchmark called AgentMafia was introduced, testing 12 advanced AI models in a series of 40 social deduction games. Researchers meticulously logged every role assignment, vote, deception, investigation, and night action. The standout performer was @Kimi_Moonshot's K3 model, which successfully identified the mafia members 72.9% of the time, significantly outperforming @claudeai's Opus 5, which detected the mafia only 48.8% of the time. This development is significant for the AI/ML community as it highlights the capabilities and limitations of current models in understanding complex social dynamics and deception. The AgentMafia benchmark not only serves as a tool for evaluating AI performance in social reasoning but also encourages further advancements in model training to improve their interpretative skills in ambiguous situations. The findings underscore the importance of robust evaluation metrics in AI research, especially when it comes to applications requiring sophisticated social understanding and decision-making.