Quantifying Context Rot (www.boolean.ai)

A recent study has quantified the phenomenon known as context rot in large language models (LLMs), specifically examining the performance degradation as context size increases during challenging coding tasks. The research found that GPT-5.6 Sol suffers a significant performance loss regardless of whether the context is task-related, with pass rates dropping from 31% to 24% at 250k tokens and remaining low at 500k tokens. Conversely, Claude Opus 5 is less affected when the context is unrelated, but experiences a 10% drop from a 50% pass rate to 40% when supplied with relevant context. The study highlights key technical considerations, including differences in tokenization between models that impact performance metrics. Notably, GPT-5.6 Sol exhibits a failure mode at extended context lengths, reverting to past Q&A responses instead of focusing on the current task, a behavior not observed in Claude Opus 5. These findings underscore the importance of context management, suggesting that new sessions or compacting contexts could help mitigate performance degradation—an invaluable insight for developers aiming to optimize LLM functionality in practical applications.