Constraint Decay: The Fragility of LLM Agents in Back End Code Generation (arxiv.org)

A recent study highlights a critical issue in the application of Large Language Model (LLM) agents for backend code generation, revealing a phenomenon termed "constraint decay." While LLMs excel at generating functional code under loose specifications, they struggle to meet structural constraints crucial for production-grade software, such as adhering to architectural patterns and database structures. The research assessed agent performance on 80 generation tasks with a unified API contract across multiple web frameworks, discovering that performance significantly deteriorates as structural requirements intensify. Strong configurations dropped an average of 30 assertion points when facing rigorous specifications, while weaker setups faltered entirely. This finding is significant for the AI/ML community as it underscores the limitations of current LLM implementations in handling the multifaceted nature of software requirements. The study shows that while LLMs can be effective in simpler frameworks like Flask, they are notably less effective in convention-heavy environments like Django and FastAPI. Moreover, key issues such as incorrect query compositions and ORM runtime violations were identified as common faults, illuminating important areas for improvement. Addressing these challenges is vital for enhancing the reliability of code generation agents, ensuring they can effectively satisfy both functional and structural demands in software development.