Why AI agents struggle with chip design (RTL) debugging, and what helps (booley.dev)

A recent experiment revealed the challenges AI agents face in debugging register-transfer level (RTL) code, specifically within the context of RISC-V processor designs like Caliptra, a hardware root of trust project. While an AI coding agent could quickly identify the faulty line of code, confirming the fix took significantly longer due to complexities in simulating the hardware environment. Over ten trials, the agent struggled with three major issues: the invisibility of internal signals unless recorded, lengthy simulation times leading to guesswork, and a common build dependency problem. These findings are significant for the AI/ML community as they highlight the limitations of current AI tools designed mainly for software debugging when applied to hardware design. AI agents typically operate under the assumption that errors produce straightforward outputs, but RTL debugging introduces unique hurdles, such as visibility of internal states and resource-intensive simulations. The study used Booley, an open-source harness customized for FPGA and ASIC work, which reduced the average time for a verified fix from 56 to 21 minutes by addressing these specific issues. This suggests that improving the AI debugging environment, particularly in terms of real-time data visibility and simulation handling, could enhance the efficiency of AI agents in hardware design.