Show HN: Fast (0.5 GB/SEC) dedup utility for the era of LLMs written in C23 (github.com)

A developer has unveiled a high-performance deduplication utility, dubbed `corpus_dedup`, designed for the burgeoning needs associated with large language models (LLMs). This tool, implemented in ISO C23, features a parallel Block Tree construction and operates at an impressive speed of 0.5 GB/sec. It scans directories for text files, processes them into manageable units (lines, sentences, paragraphs, or whole documents), and creates unique output while optionally cataloging duplicates. Notably, users have the option to build a Block Tree for further analysis, leveraging advanced hashing techniques to ensure stability and efficiency in deduplication. This utility is significant for the AI/ML community as it provides a streamlined method for handling and preprocessing vast datasets crucial for training LLMs. The ability to quickly eliminate redundant information enables researchers and developers to optimize their data workflows, thereby enhancing model training time and resource usage. Key technical features include support for assembly optimizations that can significantly boost processing speed, the capability to customize specifications such as deduplication granularity, and configurable threading options to maximize performance on different hardware setups. The project, licensed under MIT, is accessible for experimentation and integration into existing AI pipelines.