Last year we built megakernels. This year we deleted them: The road to TKCC (www.together.ai)

🤖 AI Summary
In a significant shift from its previous work on megakernels, which allowed entire model forward passes to be executed with a single highly fused GPU kernel, Together Compiler has announced a new optimization framework for AI inference that emphasizes simplicity and adaptability. After finding that megakernels, while fast, were difficult to implement in production and offered limited versatility, the research team discovered that leveraging existing technologies such as CUDA Graphs and PyTorch's Torch Inductor could achieve 98% of their performance without necessitating the creation of new kernels. Additionally, by implementing techniques like multi-kernels and agent loops, they improved performance even further, attaining a throughput of over 1100 transactions per second for the Llama 1B model on H100 GPUs. This new direction highlights the evolving complexities of AI inference, which require optimizations able to adapt to diverse model architectures and workloads. The insights gained from their research point out inefficiencies in current AI frameworks, noting that conventional compiler heuristics could overlook optimal configurations, leading to underutilization of GPU resources. By focusing on developing the Together Compiler and reevaluating assumptions about inference optimization, Together aims to provide a more effective solution moving forward, setting the stage for the next generation of efficient AI deployment.
Loading comments...
loading comments...