SWE-AGI: benchmarking spec-driven software construction (arxiv.org)

The introduction of SWE-AGI marks a significant advancement in the realm of AI-driven software development by establishing a standardized framework for evaluating the ability of large language model (LLM) agents to construct software systems autonomously from explicit specifications. Unveiled with the MoonBit framework, SWE-AGI defines a series of rigorous tasks that require LLMs to implement complex components like parsers and SAT solvers, simulating weeks of work that would typically fall to human developers. This benchmark highlights the current capabilities and limitations of LLMs in producing high-quality code based on authoritative standards, aiming to push the boundaries of AI in software engineering. The findings from initial benchmarks reveal that the gpt-5.3-codex model outperformed its competitors, successfully completing 86.4% of tasks, while challenges remain for both high-difficulty tasks and the scalability of codebases. Notably, as task complexity rises, the ability of AI models to effectively read and understand existing code becomes a critical bottleneck, indicating that while progress has been made in autonomous software engineering, substantial hurdles still exist before these systems can be relied upon for large-scale production development. This research not only underscores the current state of AI capabilities in software development but also paves the way for future enhancements and refinements in the field.