RAG Eval Comparing Vertex/Bedrock/Azure/OpenAI (github.com)

A new evaluation methodology called RetrievalCI has been introduced, providing a comprehensive comparison of hosted Retrieval-Augmented Generation (RAG) services from Vertex AI, Amazon Bedrock, Azure AI, and OpenAI. This evaluation, conducted using a consistent 50-question enterprise corpus, reveals that these hosted services outperform local open-source stacks by approximately 35 points, achieving scores between 78.5 and 84.0 while maintaining a minimal cloud expenditure of $0.17 for execution. Notably, the unique metric used in this assessment focuses on retrieval efficiency, marking a shift from traditional generation quality benchmarks prevalent in existing tools. This evaluation is significant as it offers the AI/ML community a standardized and reproducible framework for assessing the performance of various RAG systems, addressing a gap in comparative analyses that has often been presented only in vendor-specific reports. The methodology's MIT licensing and the availability of detailed technical specifications, including recall and precision metrics, enable developers to leverage these insights for deployment decisions in enterprise environments. As the community awaits further expansions—such as additional corpora and adapters—the current framework promises to guide organizations in selecting the most effective RAG solution for their needs.