From Words to Watts: Benchmarking the Energy Costs of LLM Inference (2023) (arxiv.org)

A recent study titled "From Words to Watts" presents a detailed benchmark of the energy costs associated with inference using large language models (LLMs). Conducted by researchers, this work emphasizes the often-overlooked energy demands of inference, especially as LLMs become ubiquitous in sectors like law, finance, and medicine. The paper highlights the need for improved understanding of resource utilization to facilitate cost-efficiencies, optimize hardware usage, and enhance scaling strategies in real-world applications. Using the state-of-the-art LLaMA model from Meta AI, the researchers tested inference performance across varying model sizes on NVIDIA's V100 and A100 GPUs, as well as two distinct datasets (Alpaca and GSM8K). The multi-node, multi-GPU experiments, which involved model sharding across up to 32 GPUs, represent a significant foray into analyzing the computational and energy requirements of LLM inference at a larger scale. This novel approach not only addresses a critical gap in the current AI landscape but also sets the stage for future advancements in sustainable AI practices, making it a pivotal contribution to the AI/ML community.