Show HN: Capy-flash – a 125B MoE model on two 16 GB consumer GPUs (github.com)

Capy-flash, a groundbreaking AI model, has been announced as a 125 billion parameter mixture-of-experts (MoE) architecture optimized to run on two consumer-grade 16 GB GPUs. This model, developed using NVIDIA's NVFP4 framework, showcases impressive performance benchmarks, such as generating 3,977 tokens per second with a 55k-token prompt, outperforming its counterpart Strata under similar conditions. With its unique configuration enabling efficient use of both GPU and CPU resources, capy-flash is specifically designed for coding tasks and features compatibility with OpenAI and Anthropic APIs. The significance of capy-flash lies in its ability to democratize high-performance AI model deployment, making advanced capabilities accessible to users with consumer hardware. Its efficient management of memory and resource allocation through a combination of GPU tensor-parallelism and CPU-based computations allows for enhanced performance without the need for expensive setups. Furthermore, the model employs advancements in speculative decoding and context caching, which improve response times and task accuracy. By achieving substantial token generation speed and quality metrics comparable to larger, enterprise-level models, capy-flash represents a major step towards making sophisticated AI solutions more widely available in the AI/ML community.