37% on ARC-AGI 2 without an LLM (mvakde.github.io)

A recent achievement in the field of AI/ML has emerged from the mdlARC project, where a new iteration scored an impressive 37.08% on ARC-AGI 2, despite being trained solely on ARC puzzles and not utilizing any large language models (LLMs). This score places it as one of the top submissions in the Kaggle competition, outperforming the previous year’s winning model, which relied on fine-tuning a larger 4 billion parameter Qwen model. Notably, mdlARC operates with a much smaller architecture of just 100 million parameters, showcasing the potential of efficient training strategies in deep learning. The significance of this development lies in its focus on sample efficiency and minimal data exposure, raising important questions about the dependence on LLMs for achieving high performance. The innovative use of "looping," where specific transformer blocks are executed multiple times, has proven to yield substantial score improvements. This model's unique approach includes a hub-and-spoke attention mechanism that cuts down the computation cost while maintaining performance. The results underscore a promising direction for future research: enhancing model capabilities without vast expanses of data and leveraging novel architectural tweaks that could reshape AI training methodologies.