A/B test yourself vs. code agent (arturkesik.com)

A new approach to evaluating the effectiveness of code generated by large language models (LLMs) has emerged, encouraging developers to conduct A/B tests to compare their own coding skills against those of AI. As LLMs evolve, the significant trade-offs of increased reliance on these tools are prompting developers to reconsider their deftness in coding. By assessing the quality of AI-generated code against their own creations, programmers can maintain their skills while also validating the efficiency and maintainability of AI outputs. The proposed testing method is straightforward: developers work on a feature manually without LLM assistance, logging the time taken before committing the code. After resetting their branch, they then allow an AI to tackle the same feature with minimal guidance. By comparing the outcomes—both in terms of ease of implementation and code quality—developers can gain insights into the effectiveness of AI tools. This practical experimentation not only highlights the importance of human skill development but also provides a pathway for developers to critically assess ongoing advancements in AI coding technologies.