Comparing web search API providers on a Deep Research gauntlet (www.searchspace.io)

A recent comparison of web search API providers utilizing a language model (LLM) as a judge has shed light on the effectiveness of four major players in the market. Conducted by SearchSpace, the evaluation employed a Deep Research gauntlet consisting of 32 unique tasks that simulate real-world applications rather than trivia questions. The LLM judge assessed the search outputs based on coverage, grounding, depth, and clarity, ultimately generating an ELO score for each provider based on pairwise comparisons. This approach ensures that the analysis remains objective and focused on practical use cases within high-revenue sectors such as healthcare, finance, and cybersecurity. Significantly, the study highlights the economic feasibility and latency concerns of deep research APIs, emphasizing the need for rapid response times akin to modern internet speeds for widespread adoption. The published results are made accessible via an interactive database, allowing users to explore each provider's strengths and weaknesses across various verticals. This initiative is a critical step toward establishing benchmarks for web search APIs and could influence future use cases and spending in the generative AI space, which is projected to witness substantial growth in the coming years.