Distance-Based Compression Method for Large Language Models (www.mdpi.com)

Researchers Shen and Hu have introduced a novel distance-based compression method designed to optimize memory usage in large language models (LLMs) during inference. The technique uses a trainable merge model—an MLP—that progressively compresses past token-specific cache units based on their distance from the current token, increasing compression ratios for more distant tokens until reaching a preset threshold. Crucially, the compression is deterministic, with fixed compression positions, enabling efficient inference without extra computational overhead. This approach addresses one of the critical challenges in handling long-context sequences in LLMs: managing extensive memory requirements as sequence lengths grow. By hierarchically merging cached tokens in a planned schedule, the method preserves prediction accuracy while significantly reducing memory footprint and computational cost. Experimental results demonstrate stable and quality-preserving performance across various QA and few-shot learning benchmarks, with improvements in efficiency metrics like latency, memory consumption, and energy use compared to uncompressed baselines. Technically, the strategy blends algorithmic merge planning with learned compression via an MLP, making it adaptable and scalable as model contexts extend. This contribution holds significant implications for advancing efficient long-context processing in transformers, enabling deployment of LLMs on resource-constrained systems while maintaining competitive accuracy on complex reasoning tasks.