LLMlet: P2P distributed LLM inference on browsers (github.com)

LLMlet has introduced a groundbreaking way to run large language models (LLMs) directly in web browsers, utilizing a peer-to-peer (P2P) approach for distributed inference. Built on Wasm-compiled llama.cpp, LLMlet enables browsers to connect through WebRTC using PeerJS, allowing multiple browser tabs to act as interconnected peers. Each peer can load a model, distribute its parts, and process inference requests, overcoming the memory limitations of individual browsers. This innovative setup offers a demonstration of three browser tabs interacting, showcasing real-time inference where one tab acts as a client while the others work as servers. The significance of LLMlet for the AI/ML community lies in its potential to enhance model accessibility and scalability without the need for extensive server infrastructure. By leveraging the capabilities of modern browsers, LLMlet paves the way for collaborative model usage and experimentation on a personal level. However, it does come with challenges, such as the need for sufficient resources on each peer and limitations in parallel processing, which may impact inference speed. Overall, LLMlet represents a promising step towards decentralized AI, enabling more flexible and resource-efficient model deployment.