Show HN: Bonsai 1.7B ternary model at 442T/s on M4 Max (agents2agents.ai)

A new optimized inference build of the Bonsai 1.7B model has been announced, achieving a remarkable performance of 442 teraflops per second (T/s) on Apple's M4 Max chip, primarily through the development of custom Metal kernels by an autonomous engineering agent named ata. This upgrade presents significant enhancements over the upstream Q2_0 model, which only delivered 311 T/s for decoding. The improvements are achieved without changing the model's core architecture, ensuring that the numerical output remains consistent with the reference build. This development is notable for the AI/ML community as it highlights the potential of specialized neural network optimizations on Apple Silicon, pushing the boundaries of performance achievable with existing models. Specifically, the custom-built GPU kernels target essential computational layers, showcasing a path for enhancing efficiency on hardware with varying memory bandwidth, such as the previous M1, M2, and M3 models. The Bonsai 1.7B’s efficient deployment signifies a leap in utilizing ML models in real-time applications on consumer-grade devices, underscoring the growing capabilities of AI-driven solutions in everyday technology.