Show HN: Ctrlb-decompose: Strip the noise from logs before sending to LLMs (github.com)

A new tool called Ctrlb-decompose has been introduced, designed to streamline the processing of large log files by transforming millions of noisy log lines into a concise set of actionable patterns. This tool achieves a remarkable reduction, compressing over a million lines into just 43 patterns, representing a 99.9% decrease in data complexity. Utilizing a two-stage normalization and clustering pipeline, it allows users to extract essential statistics, anomalies, and correlations from logs efficiently, either through a command-line interface (CLI), in-browser with WASM, or as a Rust library. The significance of Ctrlb-decompose lies in its ability to enhance log analysis for AI and ML applications by providing structured, token-efficient output that can be seamlessly fed into large language models (LLMs). The tool employs a unique method of normalizing logs by encoding variables as compact placeholder bytes, enabling better analysis and understanding of log data. Its features, such as categorization of log types and identification of anomalous patterns, position Ctrlb-decompose as an invaluable asset for developers and data scientists, allowing them to identify issues more effectively and optimize their workflows in log management and anomaly detection.