Show HN: AgentRuleBench, does AI agents violate inferred architecture rules? (github.com)

AgentRuleBench has launched a rigorous examination of whether AI coding agents adhere to inferred architectural rules as specified in architectural guidance documents. The study tested a specific rule regarding UI component imports within request-entry files, confirming that the AI agents from three different vendors did not violate this guideline under various experimental conditions, including a benchmarking setup that could be independently recreated. This investigation utilized archprint, an open-source tool that analyzes a repository's import graph to derive and statistically back architectural rules. This finding is significant for the AI/ML community as it addresses the ongoing debate about the reliability of AI coding agents in following architectural conventions. While the results suggest that current AI agents may not drift from established rules in this particular scenario, the study also clarifies that the scope was limited, focusing only on one specific rule within a single repository. The broader implications highlight the necessity of establishing more comprehensive benchmarks to fully understand the adherence of AI agents to diverse architectural standards in software development. For those interested, all materials, including methods and results, are publicly accessible for reproduction and further exploration.