Latent-Space Communication in Heterogeneous Multi-Agent Systems (arxiv.org)

A new framework called the Vision Wormhole has been introduced to enhance communication among heterogeneous Multi-Agent Systems (MAS) that utilize Large Language Models (LLMs). Traditional text-based communication in MAS has been inefficient due to runtime overhead and quantization loss, particularly when integrating diverse models with different architectures. The Vision Wormhole addresses this by enabling model-agnostic, text-free communication through the visual interfaces of Vision-Language Models (VLMs). It introduces a Universal Visual Codec that maps reasoning traces into a shared continuous latent space, facilitating high-bandwidth communication and effectively acting as a "universal port" for inter-agent interactions. This development is significant for the AI/ML community as it scales communication capabilities across different model families while simplifying the process from O(N^2) to O(N) for pairwise alignments. The framework uses a novel teacher-student distillation objective to synchronize the visual channel with the reasoning patterns found in traditional text pathways, allowing for reduced end-to-end processing times without sacrificing reasoning fidelity. Experimental results indicate that the Vision Wormhole performs favorably in maintaining collaborative reasoning across varied models, marking a substantial leap in the efficiency and scalability of multi-agent interactions.