Sharded Inference of a 229B-Parameter Moe over the Internet at Interactive Speed (twitter.com)

A groundbreaking technical report has been released detailing the successful execution of a 229-billion parameter mixture of experts (MoE) model across five consumer GPUs located in different countries via the public internet. The system achieved impressive interactive inference speeds of 12.6 tokens per second and 194 tokens per second for batched requests, marking a significant milestone in remote model deployment and performance. Notably, each request was embedded with cryptographic receipts, ensuring both security and data integrity for users. This achievement is particularly significant for the AI and machine learning community, as it demonstrates the feasibility of using large-scale AI models in a decentralized manner without the need for massive on-premises computing infrastructure. By leveraging consumer-grade hardware and public internet, this approach could democratize access to advanced AI capabilities, allowing more developers and businesses to integrate powerful models into their applications. The technical implications highlight a potential shift toward distributed AI, where the burden of processing can be shared across multiple locations, mitigating the challenges posed by centralized data centers.