AI Gateway Benchmark: OpenRouter, Vercel, Ramp, Keln (keln.ai)

A recent benchmark comparing four AI gateways—Keln, OpenRouter, Vercel, and Ramp—reveals significant disparities in their performance, cache efficiency, and cost-effectiveness during a 24-hour test of 13,440 turns. Keln excelled with a cache hit rate of 94%, moving provider only 14 times, while Ramp struggled with only 63% and a staggering 500 provider shifts. Vercel and OpenRouter performed moderately with cache rates of 87% and 88%, respectively. Such differences highlight how gateway architecture can drastically influence operational costs and efficiency, with Ramp facing nearly double the costs due to frequent cache misses. Furthermore, the analysis indicates that while all gateways processed turns at similar decoding speeds (around 400 tokens per second), the reasoning token outputs varied significantly. Keln's average reasoning per turn was 856, while Vercel and Ramp provided much less (403 and 420, respectively). This discrepancy not only impacts performance but also affects pricing strategies; for instance, Vercel's overall bill was $9.14 compared to Ramp's $20.20, largely due to the latter's high rate of uncached prompts. This exercise underscores the importance of gateway selection in optimizing AI service deployments, revealing that the same underlying model can yield different results based on the chosen infrastructure.