Mesh LLM: distributed AI computing on iroh (www.iroh.computer)

Mesh LLM has been announced as a new distributed AI computing solution that allows users to leverage their existing GPUs across multiple machines, presenting them as a single, OpenAI-compatible API. This approach significantly alters the conventional paradigm of large language models that typically rely on centralized data centers and costly API services. Users can start with one node and progressively expand their setup, granting greater control over model updates, privacy, and operational costs, making it particularly appealing for businesses with idle GPUs. The technical architecture of Mesh LLM utilizes iroh for seamless communication between nodes, enabling capabilities such as local execution on available GPUs, routing tasks to peers with required models, or splitting large models across several machines. This setup supports a variety of models and includes features like NAT traversal for secure peer-to-peer connections, making scalability straightforward and efficient. With the introduction of an open protocol structure, users can not only run advanced language models without massive infrastructure investments but can also create private deployments, enhancing collaboration without vendor lock-in. The upcoming mobile app further aims to integrate with the emerging ACP agent standard, broadening accessibility and functionality.