Inconsistencies in MCP support across LLMs (www.polytomic.com)

Recent investigations into the implementation of the Multi-Context Protocol (MCP) across various large language models (LLMs) highlighted significant inconsistencies in how these models handle agent output. Researchers at Polytomic discovered that certain compatibility features intended to maintain backwards compatibility resulted in duplicate data being sent to agents, complicating context consumption and potentially wasting context budget. The study used a specialized MCP server with probe tests designed to differentiate between structured and text responses, revealing that different models and their harnesses behaved inconsistently, often returning both types of data when only one was necessary. This research underscores a critical issue for the AI/ML community: even standardized protocols like MCP can exhibit substantial variability across implementations. By exposing these discrepancies through a platform called "What The Harness?", Polytomic aims to create a resource for developers to compare behaviors of various agents and versions. The findings not only indicate a need for stricter adherence to protocol specifications but also emphasize the importance of examining how these protocols are executed in practice, highlighting the broader challenge of ensuring compatibility and efficiency in AI tool integrations.