Don't trust large context windows (garrit.xyz)

A recent analysis highlights the limitations of large context windows in language models (LLMs), emphasizing that beyond a certain threshold—approximately 100,000 tokens—LLMs enter what the author describes as the "dumb zone," where their ability to retain and utilize prior information diminishes. This insight is crucial for the AI/ML community as it challenges the effectiveness of current marketing claims surrounding extended context windows that exceed 200,000 tokens or even reach into the millions. Studies indicate that the effective context is often far less than advertised, leading to performance degradation that could adversely affect coding agents' reliability during extensive tasks. The author advocates for strategies to manage context more effectively, such as using auto-compaction in tools like Claude Code, which summarizes information during lengthy sessions. However, these solutions can still result in time spent in the less effective dumb zone. A more proactive approach involves creating structured artifacts (like specifications and plans) to selectively transfer knowledge between sessions, helping maintain cognitive efficiency. By treating the context window as a budget and deliberately managing what information stays active, practitioners can enhance the utility of their LLM-driven projects and mitigate the pitfalls of context rot.