Paying for LLM inference by the kilowatt-hour instead of per token (www.coinerella.com)

NeuralWatt has introduced a groundbreaking energy-based metering system for large language model (LLM) inference, shifting the billing paradigm from traditional token-based models to pricing based on kilowatt-hour consumption. This innovative approach significantly reduces expenses for users, boasting average savings of 82.9% when compared to conventional token-based pricing structures. For instance, one of the models, Qwen3.6-35b-fast, can cut costs by up to 95.2%. Additionally, NeuralWatt's efficient caching significantly enhances performance by optimizing repeat inputs, ensuring that customers can benefit from further cost reductions. This shift is significant for the AI/ML community as it not only lowers operational costs but also aligns the consumption of AI services with energy usage, potentially encouraging more sustainable practices in computational resource management. Early users have noted some challenges, such as concurrent-request limits and occasional performance slowdowns, which suggest growing pains in adapting to this new infrastructure. Nevertheless, NeuralWatt's approach could pave the way for a more flexible, cost-effective model for LLM usage, reshaping how developers engage with AI technologies and prioritize their computational needs.