A representation model as a workaround for catastrophic forgetting in LLMs (gist.github.com)

A new architectural proposal suggests integrating a "representation model" as a workaround to combat catastrophic forgetting in large language models (LLMs). This approach aims to enable persistent self-reflection by allowing LLMs to retain insights from previous reflections, rather than resetting with each new interaction. The representation model, a smaller auxiliary network, would read the host's residual stream, create meta-vectors, and feed them back into the host, thereby facilitating ongoing behavior modifications without altering the host's weights. This design circumvents the common issue of catastrophic forgetting by ensuring the representation model is focused on higher-order properties that can be updated independently of the main model. This development is significant for the AI/ML community as it brings the possibility of genuine metacognition in LLMs, where past experiences influence future behavior. Unlike traditional methods that rely on continual training that destabilizes the model, this method allows for real-time learning by exploiting feedback from the host's predictions and errors. The innovation lies in how the components—such as logit and tuned lenses—are integrated into a feedback loop, creating a pathway for LLMs to learn from their mistakes and improve their performance in a more reliable manner. This approach opens doors to more sophisticated AI systems capable of adaptive learning, enriching their utility in real-world applications.