Per-query energy consumption of LLMs (muxup.com)

A new focus on measuring the per-query energy consumption of large language models (LLMs) is emerging, particularly for open-weight models restricted to user-hosted setups. The difficulty in obtaining reliable energy consumption figures arises from the proprietary nature of many LLMs and the variability of hardware used for their deployment. Notably, the InferenceMAX benchmark suite offers a structured approach to tackling this issue by aiming to emulate real-world applications while providing important metrics like Watt-hours (Wh) per query across various hardware configurations. The findings imply that understanding energy consumption can illuminate the economic viability of LLMs, especially as AI adoption grows. This exploration is significant for the AI/ML community as it offers a clearer picture of the environmental impact associated with deploying large models. By utilizing InferenceMAX, researchers can capture meaningful benchmarks, monitoring throughput, latency, and energy usage effectively. However, challenges remain, such as reconciling idealized benchmark conditions with practical deployment scenarios. Transparency from API providers regarding their energy statistics and query data could greatly enhance the understanding of real-world energy implications, guiding researchers and developers toward more sustainable practices in AI usage.