Do MCPs improve coding agent performance? (marginlab.ai)

A new study has examined the efficacy of Model Context Protocol servers (MCPs) in enhancing coding agent performance, with a focus on Context7, a popular MCP that offers on-demand documentation for libraries and frameworks. The research utilized Codex, running on the Terminal-Bench 2.0 benchmark, which includes 89 complex software engineering tasks. Despite the intention that real-time access to updated documentation might improve performance, the study revealed minimal impact: Codex achieved a pass rate of 70.8% with no Context7 access, compared to a slight improvement to 71.9% when Context7 was integrated, suggesting that the difference was statistically negligible. Furthermore, the investigation highlighted that Codex hardly utilized Context7, only invoking it in 6 cases out of 89, which indicates a significant reliance on its pre-existing knowledge. Notably, in all instances where Context7 was used, the outcomes were consistent with those obtained without it. This initial evaluation casts doubt on the immediate value of MCPs like Context7 for improving coding agent performance and hints at adoption challenges within AI models. Future analyses will explore additional MCPs and agents to determine if the findings hold across varied scenarios, aiming to provide more comprehensive insights into the effectiveness of these external resources.