Evaluating Long Context (Reasoning) Ability (nrehiew.github.io)

Attention around huge context windows (e.g., Grok 4 Fast’s 2M-token input) has exposed a gap: token capacity ≠ sustained reasoning ability across long inputs. The post surveys existing long-context evaluations—Needle-in-a-Haystack (NIAH), RULER, NoLiMa, LongBench, MRCR, GSM-Infinite and LoCoDiff—and argues most benchmarks emphasize retrieval (finding a needle) or trivial per-token tasks rather than the harder joint problems of relevance detection plus complex reasoning. It highlights “Context Rot,” where models that can technically access distant tokens still show degraded reasoning (hallucinations, instability, “doom loops”) as context length grows, and shows that many viral all-green plots mask this failure mode. To address this, the author proposes LongCodeEdit: a realistic, coding-focused benchmark built from BigCodeBench where syntactically valid Python functions are manually corrupted to fail provided tests. Multiple functions and their test suites are concatenated so the model must (1) scan and identify which function violates its docstring/test behavior and (2) synthesize a correct fix—evaluated with Pass@1. LongCodeEdit forces both relevance selection (every function and tests are from the same distribution) and substantive reasoning/editing across long context lengths, providing a better measure of a model’s ability to maintain quality reasoning as input size scales. The takeaway: future benchmarks should use naturally interconnected content with no “junk” tokens and measure reasoning degradation, not just retrievability.