Multi-Robot Cooperative Spatial Reasoning with Multimodal Large Language Models (arxiv.org)

Researchers have announced significant advancements in multi-robot cooperative spatial reasoning using Multimodal Large Language Models (MLLMs). In the study, titled "Seeing Together," the team introduced CoopSR, the first benchmark designed for evaluating multi-robot dynamic spatial reasoning. This involves answering complex spatial, temporal, visibility, and coordination questions through synchronized egocentric videos captured by a team of robots. The accompanying EgoTeam dataset contains over 114,000 question-and-answer pairs, allowing for diverse evaluations across varying difficulty levels and team configurations, including real-world tests with quadruped robots. The work introduces SP-CoR (Spectral and Physics-Informed Cooperative Reasoner), an innovative MLLM framework that enhances cooperative reasoning through techniques such as dynamics-aware frame sampling and physics-guided view fusion. Remarkably, SP-CoR outperformed previous models by significant margins, achieving a +3.87% improvement in the Habitat environment and +7.12% in iGibson. This advancement not only showcases the potential of MLLMs in robotics but also offers robust generalization capabilities, promising to enhance collaboration among robots in complex environments while minimizing training data requirements during real-world applications.