Show HN: GreedyPhrase – 1.21x better compression than GPT-4o tiktoken, 6x faster (github.com)

A new tokenizer named GreedyPhrase has been introduced, demonstrating enhanced capabilities over existing models like GPT-4 and GPT-4o. GreedyPhrase achieves a remarkable 1.21x better compression ratio compared to GPT-4o and 1.23x over GPT-4, all while utilizing a smaller vocabulary of just 65,536 tokens. In addition to its superior compression performance, GreedyPhrase operates 6 to 11 times faster, boasting a throughput of 47 MB/s. This tokenizer leverages a combination of phrase mining, byte pair encoding (BPE) fallback, and a Trie-based greedy encoding approach, allowing it to accurately handle gigabyte-scale datasets with zero out-of-vocabulary (OOV) errors. The significance of GreedyPhrase lies in its efficiency and scalability, making it a potentially game-changing tool for the AI/ML community, especially for applications requiring high-speed processing and data compression. The tokenizer’s ability to efficiently encode and decode large datasets in about 75 seconds positions it as a robust alternative for developers and researchers working with large corpuses. By reducing vocabulary size while maintaining high performance, GreedyPhrase enables quicker and more efficient model training and inference, paving the way for future advancements in natural language processing.