42x faster prompt lookup drafting in llama.cpp (jadidbourbaki.github.io)

A recent optimization to the llama.cpp framework has dramatically improved the speed and efficiency of prompt lookup drafting, with enhancements making it up to 42 times faster while reducing memory usage by up to 2.6 times. These performance gains stem from a series of simple optimizations influenced by the work of Daniel Lemire and Martin Ankerl. Notably, after Lemire's recent contribution, overall speed improvements have reached a staggering 140 times. This optimization is particularly significant for the AI/ML community as it enhances token generation in language models, where speed and memory efficiency are critical for real-time applications. The technical details revolve around the efficient use of n-gram caching. llama.cpp leverages three types of n-gram caches: context, dynamic, and static. The context cache updates in real-time as tokens are generated, while the dynamic cache facilitates the reuse of historical data. The static cache, built from pre-defined corpuses like WikiText-103, further optimizes the drafting process. The latest improvements included optimizing data retrieval strategies and switching certain data structures to enhance cache efficiency while minimizing latency. These changes not only increase the speed of drafting new tokens but also improve the overall memory use, ensuring that AI applications can run faster and more efficiently, thereby advancing the capabilities of language models.