Semantic Compression with Large Language Models (arxiv.org)

Researchers studied whether large language models can act as lossy semantic compressors — shrinking text and code into compact representations that preserve intent rather than every token. Using GPT-3.5 and GPT-4 via ChatGPT, they ran experiments showing LLMs can compress, store, recall and manipulate prompts and code snippets, and introduced two evaluation metrics: Exact Reconstructive Effectiveness (ERE) for literal recovery and Semantic Reconstruction Effectiveness (SRE) for preserved intent. Results indicate GPT-4, in particular, can reconstruct semantic content reliably and offers roughly a ~5× effective increase in usable token capacity compared with strict input limits. This work matters because it reframes capacity problems (short context windows, streaming or large-batch inputs) as a semantic-compression problem: instead of lossless encoding, systems can trade exact fidelity for preserved meaning to extend effective context size, improve retrieval-augmented workflows, and enable more efficient memory and summarization pipelines. Key implications include new ways to shard or stream long documents, compress chains of reasoning or prompts, and design persistent memories for agents. Limitations remain — compressed representations can introduce reconstruction errors or amplify hallucinations, and ERE/SRE benchmarking will be needed across domains and safety settings — but semantic compression opens a practical path to scale LLM use beyond fixed token limits.