Forward Self Models (jagilley.github.io)

Researchers have introduced "forward self-models," compact networks capable of predicting the activations of later layers in a neural network using earlier-layer activations. These models are significantly smaller, comprising only 1-3% of their main counterparts, yet they achieve up to 97% cosine similarity with target layer activations and effectively recover up to 94% of a layer's contribution to output distributions. By observing these predictions, forward self-models enhance mechanistic interpretability and provide insights into the computational dynamics of larger models. The significance of this development lies in its potential to bridge gaps in our understanding of how complex neural networks, particularly transformers, process information. Forward self-models operate as passive observers, learning from prediction errors that highlight aspects of the computation which are difficult to compress. This offers valuable metrics on the computational complexity each layer contributes, laying groundwork for future architectures that may rely on explicit modeling of their computations. With code available for widespread access, this innovation could inspire new methods for monitoring and improving model performance in AI/ML applications.