Meta Superintelligence's surprising first paper (paddedinputs.substack.com)

Meta Superintelligence Labs’ first paper, REFRAG, isn’t a new model architecture but a systems-level RAG optimization that converts most retrieved document chunks into compact, LLM-aligned chunk embeddings the LLM can consume directly. Documents are chunked (~128 tokens), pre-encoded into compact embeddings and projected into the LLM embedding space. At inference a lightweight policy network (trained with an RL objective to minimize downstream perplexity under an expansion budget) chooses a few chunks to expand back into full tokens; the LLM receives a mixed input of a short token sequence (query + expanded chunks) plus single-vector placeholders for unexpanded chunks and generates as usual. The result: much less KV-cache and attention overhead, hugely reduced time-to-first-token (MSI reports up to ~30x faster TTFT), and higher throughput while preserving perplexity and benchmark accuracy. This is strategically significant because RAG is widely deployed and inference cost/latency directly determine UX and product ROI; software-level gains buy headroom without new GPUs. REFRAG is orthogonal to better retrievers/rerankers and can be combined with them, but it does add engineering and training steps (encoder + projection, reconstruction/SFT, an RL-trained selection policy), has a compression-accuracy ceiling, and requires refresh pipelines for dynamic corpora. It’s an enabling efficiency play — not magical reasoning — that could shift operational costs and inspire further “embedding-native” read/write designs for agents and retrieval-heavy apps.