Cascadia: Split LLMs Across Intel PCs (github.com)

Cascadia has introduced an innovative framework that enables distributed inference of large language models (LLMs) on Intel hardware, allowing users to shard models across multiple PCs without relying on cloud services or NVIDIA GPUs. This significant development addresses the challenges posed by frontier models that typically exceed the capacity of individual devices, as well as the high costs and data privacy concerns associated with cloud APIs. By offering an OpenAI-compatible API, users can integrate existing clients seamlessly. Key technical features of Cascadia include a built-in sharder for creating INT4 per-stage shards from HuggingFace models, and pipeline parallelism that distributes different model stages across available machines. The framework also supports peer discovery for easy network configuration, ensuring simple setup on Intel AI PCs with various GPU architectures. With its alpha release, Cascadia simplifies the deployment of advanced LLMs, promoting more accessible AI model inference for developers while advancing the capabilities of CPU-based AI workflows.