CAVEWOMAN: How LLMs Behave Under Linguistic Input and Output Compression (arxiv.org)

Researchers have introduced "Cavewoman," a new evaluation protocol that explores how Large Language Models (LLMs) respond to different types of linguistic input and output compression. The study reveals that while compressing the output of LLMs can significantly reduce inference costs—by up to 3x in some cases—compressing the input leads to increased costs and decreased accuracy. This dual-channel assessment analyzes eight models across five datasets, highlighting a net cost increase of approximately 1.15 times when input compression is utilized, as models tend to generate longer responses with reduced accuracy. The implications of these findings are critical for the AI/ML community, particularly for those looking to optimize costs while maintaining performance. The research challenges the prevalent notion that simplifying input prompts will yield efficiency gains, instead demonstrating detrimental effects on output quality. The available code and data encourage ongoing experimentation, providing an opportunity for further exploration into the cost-efficiency balance of LLMs, which is crucial for developing scalable AI applications in real-world contexts.