From Golden Gate Bridge to JSON: Why Anthropic's SAE Failed on JSON Output (huggingface.co)

A recent investigation revealed that Anthropic's activation steering technique, intended for AI safety, is ineffective for generating valid JSON outputs, despite initial hopes that it could enhance structured data generation in large language models (LLMs). The author conducted a series of experiments, finding a stark decrease in valid JSON generation—from 86.8% with an untrained model to just 24.4% using steering techniques. This study underscores the critical challenge of ensuring high-quality output in real-world applications, where even small failure rates can lead to significant operational issues, such as API failures and data corruption. The technical implications are profound: activation steering modifies internal model behavior by adjusting neural activations, yet it proved detrimental when applied to JSON generation. The steering vectors failed to improve structural accuracy, reflecting a fundamental misunderstanding of how LLMs process syntax versus semantics. While fine-tuning the model significantly improved JSON output quality to 96.6%, all steering variants led to performance degradation, highlighting that techniques effective in one domain—like bias modification—may not translate to others involving complex output structures like coding or formatting. This exploration calls into question the broader applicability of steering methods for task-specific operations and emphasizes the necessity for robust training approaches for structured data generation.