Backdooring Sparse Autoencoders (arxiv.org)

Researchers have revealed a vulnerability in sparse autoencoders (SAEs), components often used to enhance and interpret language models. In a new study, the authors demonstrate how a backdoored SAE can be maliciously embedded into a language model's architecture to trigger predetermined behaviors, while the main language model remains unchanged. This represents a critical supply-chain attack threat, as it allows for unwanted behavior—such as unsolicited code generation—to manifest through specific prompt cues without altering the core underlying model. The significance of this work lies in highlighting that SAEs, while typically seen as auxiliary tools, can possess security vulnerabilities that could compromise the integrity of AI systems. The study's findings show that the effectiveness of these backdoored components can vary across different language models and insertion layers, making their detection and mitigation complex. This research urges the AI/ML community to reevaluate the security implications of such auxiliary components, as they can carry substantial risks without overt modifications to the primary algorithms or models involved.