Do Stronger Type Systems Make Coding Agents Better? (fedur.co)

A recent experiment explored whether stronger type systems in Python improve the performance and accuracy of AI coding agents, specifically Claude and Codex. Over 766 runs, the agents worked with various code versions—ranging from untyped to fully typed—with the goal of assessing if type annotations could enhance coding efficiency, reduce errors, or utilize fewer tokens. Surprisingly, the agents achieved near-perfect accuracy across all code types, but there was no observed increase in speed or token economy when using type annotations. The only factor that had a significant positive impact was incorporating mypy, a type checker, which eliminated type errors consistently. This study is a significant contribution to understanding how AI systems interact with programming languages. It underlines that, while type systems did not enhance the agents’ performance as hoped, the presence of a type checker drastically improved code quality. This insight suggests that AI coding agents benefit more from robust checking and feedback mechanisms rather than solely from the use of typed languages, challenging the assumption that stronger type systems inherently lead to better AI code generation.