Does Speaking to Agents Like Cavemen Save 65% of Tokens? We Test (blog.jetbrains.com)

JetBrains conducted a test on the "Caveman" skill in the Claude Code AI model to evaluate its claims of achieving 65% reduction in output tokens during coding tasks. Using an A/B benchmark approach, they forced the skill's activation to measure its effects on token saving and output quality. The findings revealed that while the skill could compress token usage, the actual savings were around 8.5%, largely due to the format of responses dominated by tool calls and code rather than conversational output. Moreover, the quality of the generated content remained statistically consistent across tasks, indicating no detrimental effects from the style change. The significance of this experiment lies in its implications for developers relying on AI coding agents. While "Caveman" can streamline communication, the expectation of substantial cost efficiency is overstated, with real-world applications yielding modest benefits. This nuanced understanding encourages users to approach such skills with realistic expectations, leveraging them for improved engagement without fearing deterioration in output quality. The test serves as a reminder of the importance of validating performance claims within the rapidly evolving AI/ML landscape.