Looking for contributors – trie based memory efficient LLM runner (github.com)

A new initiative has emerged in the AI community, announcing the SALT (Salience-Aware Lexical Trie) framework, which enables memory-efficient operation for language models (LLMs). The core of SALT lies in its innovative memory management system for handling extensive text inputs. By forming a trie structure from document keywords, SALT allows for nuanced theme-based selection of sentences, helping to avoid a common issue known as "theme collapse" where significant but minor themes are overlooked. This method not only compresses long documents to fit within a token budget but also enhances the efficiency of conversation memory by carrying forward previously selected relevant information across turns. This development is significant as it addresses the memory challenges faced by LLMs dealing with lengthy conversations or documents, effectively improving their performance and responsiveness. The SALT architecture features a dual-phase processing model, where the first phase indexes the keywords for efficient retrieval, and the second maximizes theme coverage during selection using a lazy-greedy algorithm. As a result, it optimizes the use of memory resources and computational power, reducing the cognitive load on users and making dialogue with AI more seamless. The project is open for contributions, fostering collaboration within the AI/ML community to explore further applications and improvements in conversational AI technology.