GPU-accelerated context memory for on-device AI agents on Apple Silicon (github.com)

A new framework called ContextCore has been launched to enhance on-device AI agents on Apple Silicon, specifically targeting issues related to Large Language Models (LLMs) forgetting context during extended interactions. By utilizing Metal compute shaders, ContextCore allows for the building of context windows in under 5 milliseconds, significantly improving response times and efficiency during conversations. This framework features four distinct memory tiers—working, episodic, semantic, and procedural—each with tailored retention and retrieval rules, ensuring that relevant information remains accessible without wasting tokens on irrelevant history. The significance of ContextCore lies in its GPU-accelerated technology, which processes up to 63 million chunks per second, effectively outperforming traditional CPU approaches. Key functionalities include progressive compression of low-signal chunks when token budgets are tight and attention-aware reranking, ensuring that the AI focuses on the most pertinent content first. This move not only optimizes the performance and memory management of intelligent agents but also provides developers with flexible configuration options, enhancing their ability to customize the AI's memory and context capabilities using the framework in Swift-based applications on various Apple platforms.