LongCLI-Bench: Benchmark and Study for Long-Horizon Agentic Programming in CLIs (arxiv.org)

Researchers have introduced LongCLI-Bench, a new benchmark for assessing long-horizon agentic programming capabilities within command-line interfaces (CLIs). Previous benchmarks fell short, focusing on short tasks and suffering from data contamination and vague metrics. LongCLI-Bench features 20 meticulously curated tasks derived from real-world workflows and computer science assignments, spanning categories such as bug fixing and refactoring. It utilizes a dual-set testing protocol to evaluate both requirement fulfillment and regression avoidance while incorporating step-level scoring to identify specific execution failures. This benchmark is significant for the AI/ML community as it addresses critical gaps in evaluating the actual performance of AI agents in realistic programming scenarios. Initial experiments revealed that even state-of-the-art agents struggled, achieving pass rates below 20%, with many tasks stalling early in the execution process. While self-correction methods showed minimal improvement, the study highlighted that enhanced collaboration between human users and agents can lead to substantial gains. The findings emphasize the need for future research to focus on developing synergistic workflows that leverage both human insight and AI planning capabilities, ultimately improving the efficacy of long-horizon task execution.