Run large language models at home, BitTorrent‑style (petals.dev)

A new project called Petals has emerged, allowing users to run large language models (LLMs) like Llama 3.1, Mixtral, Falcon, and BLOOM directly from their homes using a decentralized BitTorrent-style network. This innovative approach enables individuals to fine-tune and generate text with models that can reach sizes up to 405 billion parameters, all on consumer-grade GPUs or platforms like Google Colab. Users can load parts of the model while connecting with others to access the remaining segments, facilitating efficient single-batch inference speeds of up to 6 tokens per second for certain models, making it viable for real-time applications like chatbots. The significance of Petals lies in its departure from traditional LLM API reliance, offering users unprecedented flexibility in model training and deployment. Users can take advantage of various fine-tuning techniques, customize model pathways, and explore hidden states, blending the ease of API usage with the robustness of PyTorch and 🤗 Transformers. This project, part of the collaborative BigScience research initiative, exemplifies a shift towards democratizing access to powerful AI tools, empowering developers and researchers to leverage state-of-the-art LLMs more creatively and effectively.