Show HN: Coding agents find the right GPU bottleneck 70% of the time, fix it 30% (ayushnangia.github.io)

A recent evaluation of three open-source models—MiniMax-M2.1, GPT-OSS-120B, and GLM-4.7—revealed significant challenges in their ability to optimize GPU bottlenecks effectively. The investigation highlighted that while these agents could generate detailed plans (MiniMax) or make valid code edits (GLM), none managed to execute these optimizations successfully. MiniMax generated over 81,000 output tokens without executing any tool calls, while GPT-OSS attempted to create mock library implementations instead of optimizing existing code, illustrating a clear lack of understanding of the task environment. GLM-4.7 succeeded in making code changes but failed to complete the workflow due to misinterpretation of error messages related to code patches. These findings underscore critical shortcomings in current AI/ML models' tool use, environmental understanding, and workflow management, which are essential for real-world applications. The inability to effectively integrate these functionalities hampers the potential of AI agents to assist in practical coding tasks, particularly in optimizing complex systems. As the AI/ML community continues to push the boundaries of machine learning capabilities, addressing these specific failings could pave the way for more effective and reliable coding assistants in the future.