Show HN: TokenMeter – Open-source observability layer for LLM token costs (github.com)

TokenMeter has launched as an open-source AI Cost Intelligence Platform designed to monitor and optimize the costs associated with large language model (LLM) API usage. By acting as a proxy for all API calls, TokenMeter tracks every token spent, providing real-time visibility into costs per developer, model, and feature. Key features include smart routing to select the most cost-effective model, a semantic cache that can reduce token usage by up to 80%, and tools for waste detection that identify inefficient model use and failed requests. The platform also boasts a prompt optimizer, an ROI tracker, and robust budget management tools. This development is significant for the AI/ML community as it addresses the growing need for transparency and cost control amidst rising API service costs. TokenMeter's self-hosted nature ensures that user data remains private while providing extensive integration capabilities, including webhooks for alerts and the ability to manage budgets effectively. With backend support in Python (FastAPI), a React-based frontend, and a compact SQLite database, it can be easily set up and customized. As organizations increasingly rely on LLMs, tools like TokenMeter could lead to more efficient usage and management of resources, ultimately facilitating broader adoption of AI technologies.