Beating GPT-2 for less than $100 – Andrej Karpathy (github.com)

Andrej Karpathy has achieved a remarkable breakthrough by developing a method to outperform OpenAI's GPT-2 for under $100. By utilizing a single 8×H100 GPU node and approximately three hours of training, he has significantly reduced the cost of surpassing GPT-2's performance by nearly 600 times. This reduction is emblematic of the rapid advancements in AI training capabilities, with costs expected to decrease by around 40% each year. The improvements stem from several factors, including enhanced hardware, innovative software techniques like Flash Attention 3, and novel algorithms such as the Muon optimizer. The implications of this achievement are crucial for the AI/ML community, as it opens up new avenues for training complex models at a fraction of the previous costs, democratizing access to advanced AI technologies. Karpathy's work, which aims to meet the CORE metric from the DCLM paper, also includes a new leaderboard to track progress in achieving state-of-the-art capabilities against GPT-2. The focus on resource-efficient machine learning models not only fosters innovation but also highlights the potential for further scaling down model training in various applications, encouraging experimentation within the research community.