I trained a small transformer in 1.5hrs and it beats many LLMs (mvakde.github.io)

A researcher has successfully trained a small transformer model from scratch in just 1.5 hours, achieving performance levels that surpass many large language models (LLMs) and matching existing models like TRM and HRM. This development is noteworthy for the AI/ML community as it addresses the critical issue of sample efficiency—an area deemed vital for advancing AI capabilities. The model's performance on the ARC-2 benchmark, scoring 7%, highlights its ability to solve complex problems with minimal data, using only 1,000 puzzles, thereby underscoring its efficiency in a high-dimensional space. Technically, the model incorporates several innovations, such as full autoregressive training on task-specific inputs and using a new architecture with improvements like SwiGlu activation functions and advanced attention mechanisms, which allow for better sample utilization and reduced training costs. The researcher aims to encourage widespread participation in AI development by maintaining an open-source approach, allowing others to build on this work. The focus on sample efficiency and cost-effective training methods could pave the way for breakthroughs in AI research, making advanced model training more accessible even to those with limited resources.