Shaping and Monitoring Models Through Their Internal Representations (www.goodfire.com)

Recent research has introduced a novel approach to training AI models called mechanistic shaping, which addresses the common issue of models learning to achieve high scores for the wrong reasons. For instance, a model tasked with diagnosing pneumothorax may rely on confounding features, like the presence of a chest drain in X-ray images, rather than understanding the actual medical condition. This new framework employs two techniques: reliance shaping, which reduces dependence on misleading features, and expression shaping, which penalizes the model for expressing unwanted behaviors. This approach aims to improve model reliability and interpretability, especially as AI capabilities evolve and shortcuts become more complex. The significance of this work lies in its potential to align AI models more closely with human judgment and intention. By specifically guiding models to focus on meaningful internal representations, such as the features of a lung instead of confounding identifiers, the researchers show promise in enhancing diagnostic accuracy in medical imaging and reducing undesired behaviors in language models, such as sycophancy and reward hacking. The results indicate that mechanistic shaping not only helps in crafting better-performing models but also maintains their monitorability, which is crucial for maintaining ethical and functional AI systems in realistic scenarios.