Language Modeling, Part 5: Reverse Engineering LSTM Cells (connorjdavis.substack.com)

In the latest installment of a language modeling series, the author delves into the internal workings of a trained LSTM (Long Short-Term Memory) model, focusing on reverse engineering its hidden units to understand their specific functions. The author trained a 1-layer LSTM on the TinyStories dataset, which generates coherent yet imperfect stories, revealing that while the model demonstrates a grasp of syntactic structure, it struggles with some semantic coherence. The key aim of this analysis is to identify which specific hidden units are responsible for particular features, such as recognizing punctuation and quoted text, using activation traces from the model. The significance of this exploration lies in its potential to enhance our understanding of LSTM cells and improve language model accuracy. By visualizing activation patterns, the analysis uncovers that certain cells exhibit strong responses to quotes and punctuation, indicating a form of internal state representation. Further, the study examines the effects of manipulating specific cell activations to limit quote generation without degrading overall model performance. Initial results show that clamping the activation of a cell associated with quotes can significantly reduce the number of quotations in generated stories while even marginally improving perplexity, showcasing a novel approach to fine-tuning capabilities in AI language models.