Benchmarking AI gateways At 10000 RPS (vidai.uk)

A performance benchmarking study has been conducted on various AI gateways, emphasizing the significant challenges of latency as enterprises transition to production-scale generative AI workflows. Dubbed the "Performance Tax," this latency can drastically reduce user engagement. The researchers developed VIDAI, an AI gateway designed to be "invisible," providing essential functionalities without becoming a bottleneck. In the study, their Rust-native engine was benchmarked against popular gateways including Bifrost (Go), LiteLLM (Python), and Portkey (NodeJS). The results indicated a staggering performance gap, showcasing VIDAI's ability to handle 10,000 requests per second (RPS) with much lower latency compared to the competitors, particularly under high-load conditions. This study highlights crucial architectural implications for the AI/ML community. Key findings reveal that while both VIDAI and Bifrost maintained sub-50ms p95 latency at 10,000 RPS, Rust's efficient memory management significantly outperformed the garbage-collected environments of other languages, which experienced noticeable overhead. The research also identified the functional trade-offs between safety features and performance; VIDAI implemented robust features like authentication and telemetry with minimal impact on latency, illustrating its superior design for production environments that demand both high throughput and low latency. This positions VIDAI as an optimal choice for teams requiring seamless and efficient agentic workflows.