Cohere's North Mini Code Megakernel Serving Engine (cohere.com)

Cohere has unveiled its new serving engine for North Mini Code, a 30 billion parameter model optimized through a decode megakernel that significantly improves efficiency. By processing the entire forward pass as one continuous operation rather than a series of smaller kernel launches, this innovative approach achieves a decoding speed of up to 292 tokens per second at batch size 1, which is 1.58 times faster than the previous vLLM implementation. The engine’s design effectively leverages the H100 GPU’s bandwidth, showcasing that with reduced launch and synchronization overhead, it can operate at 62% of its theoretical maximum speed, demonstrating how memory bandwidth can be utilized more effectively. This development is significant for the AI/ML community as it represents a major leap in the efficiency of inference engines, particularly for memory-bound tasks like autoregressive decoding. Cohere’s serving engine addresses previously untapped potential in GPU performance by minimizing idle time and false dependencies, enhancing the throughput of operations without sacrificing accuracy. With simplified implementation processes for megakernels, Cohere is paving the way for faster, more scalable AI applications, while also sharing their findings on GitHub to spur further innovation in the field.