J-Space Oddity: Do VLMs Dream of Text Tokens? (ykumar.me)

Anthropic has published compelling research exploring the interpretability of vision-language models (VLMs) through a novel technique they call the Jacobian lens (J-lens). This approach examines how minor changes in the model's residual state impact its final output, effectively creating a multi-dimensional "workspace" called J-space. The experiments, primarily conducted with text-only prompts except for one ASCII art image, aimed to determine if such a method could also be applied to VLMs like Qwen2.5-VL, revealing how visual inputs merge into the model's text-centric processing pipeline. The significance of this research lies in its deep dive into the inner workings of VLMs, suggesting that these models inherently communicate in textual terms despite their visual components. By computing and analyzing the Jacobian matrices across various layers, the researchers uncovered that the top-scoring words in the J-space often skewed towards Chinese, reflecting the model's pretrained data distribution. The findings raise important questions about the representational framework of VLMs and their interpretative capabilities, suggesting that while these models can accurately identify visual elements (like locating an object), subtle factors such as prompt phrasing can drastically affect the outcomes, indicating an area ripe for further exploration in AI interpretability.