Show HN: Ctoken – CLI util to count LLM tokens in files, dirs or input (github.com)

Ctoken is a newly released command-line utility that simplifies the task of counting tokens in files and directories, useful for developers working with large language models (LLMs) and AI agents. By providing a concise token count summary for various file types within a directory, it helps developers understand how context windows will be affected when submitting data to coding agents. The tool supports features like grouping by file type, pattern filtering, and various output formats including JSON, making it a versatile addition to developers' toolkits. This utility is particularly significant as it streamlines a previously cumbersome process—estimating token counts using size or word counts was inefficient, especially for repeated tasks across multiple files. Ctoken leverages the tiktoken-rs library for accurate token estimation compatible with different LLMs, thereby facilitating easier comparisons of file sizes in terms of tokens. It also features a customizable profile system for different programming languages, allowing users to tailor the tool to their specific needs. Ctoken not only enhances productivity but also ensures that developers can make informed choices about context management in AI applications.