Show HN: I built a token-per-second counter in Go (github.com)

A developer recently announced the release of a new command-line interface (CLI) tool, "tokps," written in Go, that measures token-generation throughput (TPS) for any OpenAI-compatible /chat/completions endpoint. This tool is significant for the AI/ML community as it provides a standardized way to evaluate the performance of various AI models, including OpenAI's offerings and other local or custom implementations. By sending a test prompt with streaming enabled, tokps tracks the rate of token generation and reports detailed metrics such as the median time to first token (TTFT) and output tokens. Key technical details include the ability to run multiple benchmarks with configured warm-up periods to minimize the impact of cold starts and network variability. The tool outputs essential statistics such as the median TPS and observed ranges, which can be critical for fine-tuning model deployments and ensuring responsiveness in production environments. Additionally, it supports concurrent requests to test throughput under load, making it a valuable resource for developers looking to optimize and compare the performance of their AI systems. By offering a machine-readable JSON output, tokps also facilitates integration into continuous integration (CI) workflows for ongoing performance monitoring.