Mindstorms in Natural Language-Based Societies of Mind (arxiv.org)

Researchers propose and experimentally validate "natural language–based societies of mind" (NLSOMs): collections of heterogeneous neural networks (LLMs plus multimodal and specialist NNs) that solve problems by interviewing one another in a shared natural-language protocol — a "mindstorm." Drawing on Minsky’s society-of-mind and Schmidhuber’s learning-to-think, the authors show that using natural language as a universal symbolic interface makes agent composition modular and extensible; they assemble NLSOMs with up to 129 members and demonstrate improved multimodal zero-shot reasoning across tasks such as visual question answering, image captioning, text-to-image and 3D generation, egocentric retrieval, embodied AI, and general language-based problem solving. The work is significant because it frames large-scale, mixed-architecture AI as social systems rather than monolithic models, enabling task-specific experts to be plugged in, queried, and coordinated via interpretable dialogue. Key technical implications include modularity for rapid experimentation, enhanced multimodal reasoning through inter-agent debate/consultation, and the potential to scale to much larger societies (even involving humans). The paper also raises new research directions: optimal social structures (hierarchical vs democratic), incentive and reward allocation in reinforcement-learning NLSOMs, and the economics of neural-agent collaboration — all critical for designing robust, scalable, and aligned multi-agent AI systems.