One Layer, +12%: What 667 Configs Reveal About Small LLM Anatomy (austinsnerdythings.com)

A recent exploration into the structure of small language models (LLMs) has revealed that simply rerunning specific middle layers during inference can lead to significant improvements in output quality. This technique, known as Repeating Your Stack (RYS), first outlined by David Noel Ng on larger models, has been tested on a 4-billion-parameter model, Qwen3-4B. By selectively repeating layers, Ng found up to a 15.6% boost in reasoning performance on a 27-billion-parameter model. The current investigation confirmed that a three-phase operational anatomy—encoding, reasoning, and decoding—also exists at the smaller scale, with notable performance gains achieved by repeating middle layers. The findings suggest that smaller models exhibit less specialized layers, allowing single-layer repetitions to produce meaningful enhancements. Specifically, a single pass through layer 21 yielded an 11.9% improvement with minimal latency overhead, marking a practical revelation for deploying smaller dense models effectively. This research not only offers insights into optimizing LLMs for better performance without retraining but also poses implications for future AI framework developments, suggesting that layer duplication should be integrated as a standard inference feature across various model implementations.