DroPE: Extending the Context of LLMs by Dropping Their Positional Embeddings (pub.sakana.ai)

A new method called DroPE (Dropping Positional Embeddings) has been introduced to extend the context limits of language models (LMs) without the need for extensive fine-tuning or long context data, addressing a significant challenge faced by AI systems in handling long texts. Traditional transformer models use positional embeddings (PEs) to establish token order, but when these models are pushed beyond their trained context lengths, the PEs can cause performance degradation due to mismatched positions. DroPE circumvents this issue by removing the PEs after pretraining and performing a short recalibration, allowing models trained on short contexts to effectively process much longer sequences while maintaining high accuracy. This approach is notable for its efficiency, as it can be applied to various model sizes—from smaller models to those with billions of parameters—and requires minimal additional computational resources. In experiments, DroPE consistently outperformed other scaling techniques and long-context architectures, significantly enhancing performance on tasks like summarization and question answering with long content. By demonstrating that recalibrating language models post-pretraining can lead to robust zero-shot context expansion, this innovation holds great promise for advancing the capabilities of AI systems in real-world applications involving extensive and complex text processing.