Ekka: Automated Diagnosis of Silent Errors in LLM Inference (syfi.cs.washington.edu)

Ekka, a novel automated diagnosis tool, has been developed to tackle silent errors in large language model (LLM) inference, which can degrade output quality without generating visible error messages. By employing a method known as differential debugging, Ekka compares the execution states of potentially faulty frameworks against trusted references to identify precise divergences. In tests involving 17 real-world bugs from frameworks like vLLM and SGLang, Ekka achieved an impressive 80% diagnosis accuracy at a cost of approximately $30 per case and even discovered four new bugs that developers subsequently confirmed. The significance of Ekka lies in its ability to streamline a typically tedious and time-consuming manual diagnosis process that can take months. Traditional debugging methods often rely on toggling configurations and inspecting activations, which are cumbersome and error-prone. Ekka automates this by mapping equivalent components across frameworks, aligning outputs for comparison, and analyzing errors with a focus on precision. This approach reveals that silent errors often have complex underlying causes, making it essential for tools like Ekka to be model-aware. As LLM technologies progress, Ekka's framework could minimize the impact of silent errors on model performance and enhance the reliability of AI systems. Future developments may include expanding its capability beyond PyTorch and improving efficiency in diagnosis processes.