Social Intelligence Benchmark (gertlabs.com)

A groundbreaking advancement in AI evaluation has been announced with the launch of the Social Intelligence Index, designed to measure how well AI models make decisions in social multi-agent environments. Unlike traditional benchmarks that focus on coding performance through game simulations, this new index assesses models in complex, interactive scenarios where communication and social dynamics play crucial roles. This innovation emphasizes “theory of mind” over mere coding skill, allowing for a richer understanding of general intelligence. The Social Intelligence Index ranks frontier models based on their performance in decision-making environments, such as customer service simulations where agents must deduce the intentions of callers—potential identity thieves or legitimate customers. Notable models like Gemini 3.5 Flash and Claude Sonnet 4.5 lead the index, achieving high success rates through their innovative strategies. The implications extend beyond gaming simulations; the ability to understand and respond to nuanced human intentions is essential for developing robust AI systems capable of effective communication and task delegation in real-world applications. This shift in focus towards social intelligence could significantly impact the evolution of AI towards more sophisticated, human-like understanding.