The Optimal Architecture for Small Language Models (huggingface.co)

A recent study explored the optimal architecture for small language models, training 19 different configurations across 12 architecture families using 1 billion tokens. The key finding revealed that models segregate into two distinct performance tiers, with a significant threshold for hidden dimensions at 512. The research identifies a "Goldilocks" configuration of 32 layers as ideal, slightly outperforming the standard 12-layer setup at the same parameter count while also achieving superior natural language processing performance on various benchmarks. The study produced the Dhara-70M model, a diffusion architecture that is 3.8 times faster and boasts the highest factuality scores among all tested models, despite sacrificing a small portion of accuracy. By employing the Warmup-Stable-Decay (WSD) method, researchers efficiently converted an autoregressive model (LLaMA3-Canon) to diffusion with substantial savings in training time and resources. The findings underscore the significance of architectural choices and innovative training techniques in enhancing the efficiency and efficacy of small language models, setting a new benchmark for natural language processing applications.