Words that make language models perceive (www.sophielwang.com)

Researchers have uncovered new insights into how language models can be influenced to "perceive" by incorporating sensory prompts, despite operating primarily through text-based inputs. Their study reveals that when a language model is prompted to imagine sounds or visuals, the neural representations generated during text output become aligned with those of corresponding sensory modalities. This phenomenon indicates that models can be actively steered at inference time, enhancing their capacity to generate contextually rich descriptions that evoke visual or auditory imagery. This work challenges existing notions that representations in language models are fixed, instead suggesting that alignment can be manipulated through specific prompts. By employing a method called mutual k-nearest neighbors (mutual-kNN) to measure representation similarities, the researchers demonstrated that sensory cues effectively increase alignment with visual or auditory information compared to outputs generated without such cues. The findings hold significant implications for the development of more advanced multimodal AI systems, offering avenues for enhancing generative capabilities by integrating sensory modalities more effectively into machine learning frameworks.