Seeing Is Free, Speaking Is Not: Uncovering the True Energy Bottleneck in Edge (arxiv.org)

A recent study has unveiled critical insights into the energy consumption of Vision-Language Models (VLMs) during on-device inference, challenging longstanding assumptions about energy efficiency in AI applications. Researchers conducted a comprehensive energy profiling of VLMs across various architectures and resolutions on two hardware platforms, revealing that the power draw remains largely constant regardless of input conditions. They found that the encoding of output tokens incurs significantly higher energy costs—11 to 39 times more than input tokens—making output length a major factor in both latency and energy consumption. This research has significant implications for the AI/ML community, especially for the development and deployment of AI applications on resource-constrained edge devices. The findings suggest that strategies focused solely on reducing visual tokens may yield minimal energy savings, while controlling output lengths can potentially lead to reductions of up to 97% in total energy usage for larger models. By highlighting the true bottleneck in edge VLM inference as the generation of output rather than visual processing, this study encourages a reevaluation of energy efficiency strategies in AI, ultimately guiding more sustainable practices in edge computing.