Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (arxiv.org)

Recent research has highlighted the need for inference-efficient large language models (LLMs) in light of their increasing deployment and the rising costs associated with their use. Traditionally, scaling up the number of parameters and training data has enhanced model performance, yet there has been little focus on the delicate balance between accuracy and inference efficiency. This study introduces a conditional scaling law that integrates architectural considerations, examining how factors such as hidden size, the ratio of MLP to attention parameters, and grouped-query attention (GQA) impact both inference costs and model accuracy. The researchers trained over 200 models with parameter sizes ranging from 80M to 3B and training tokens from 8B to 100B, leading to the development of a framework for identifying architectures that maximize efficiency and effectiveness. The findings indicate that the proposed conditional scaling law accurately predicts optimal design choices, resulting in models that not only surpass established open-source baselines but also achieve up to 2.1% higher accuracy and 42% greater inference throughput compared to existing models like LLaMA-3.2. This advancement holds significant implications for the AI/ML community, as it opens up new avenues for designing LLMs that are both powerful and resource-efficient, addressing a critical challenge in the deployment of advanced AI systems.