Show HN: Kairo – Fail-closed LLM inference routing from RTX 5090 measurements (github.com)

Kairo, a new AI research workbench, has been developed to optimize large language model (LLM) inference routing using evidence-driven analysis on NVIDIA's RTX 5090 GPU. Through the implementation of CUDA Graphs, Kairo has demonstrated the ability to enhance NVFP4 serving throughput by over 2x for certain workloads, significantly outperforming traditional eager execution methods. The innovation lies in Kairo's methodology of capturing the specific conditions of measured workloads and employing a fail-closed runtime policy that ensures correctness by only promoting well-defined and validated route selections. This development is significant for the AI/ML community as it lays the groundwork for more efficient model serving architectures, moving beyond generalized inference engines. Kairo focuses on precise workload definitions and corrective measures while recording key metrics such as cache behavior and execution timing. Although still in the research phase, Kairo aims to contribute to the evolution of native Blackwell inference components, ensuring rigorous standards of correctness and performance measurement, promising a more streamlined process for deploying AI models in production environments.